:W&Bá¾Å™/kM 6affo2jbª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0·º\ú‚" 3nludrfi4t27 6affo2jbŠÖ 6affo2jbjuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.10441075593820484 epochs‚10  hidden_dim‚128 ( learning_rate‚0.00015404012739761552  num_attention_heads‚8  num_branch_encoder_layers‚5  num_embedding_layers‚6  num_encoder_layers‚6Bjjtttt_vanilla_6affo2jbb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8Š ÚÝåÔØ˲jš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64Ÿ‡7 6affo2jb&  _runtime‚0  _wandb.runtime‚0ó>t'iÒã -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ÙÝåÔØÀ¤b*-of*=/data/spatop/options_files/sweep_tttt_jj/config_6affo2jb.json*-l* /scratch/logs*-n*jjtttt_vanilla_6affo2jb2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8r/opt/conda/bin/pythonº 5wlbbsdhrfxquyayudxnkmru1md8tkc8†<ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€À…ê‹ ª€àãÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-78f24734-e3a1-59a7-ce1b-bb4283f77c97Ê13.2º 5wlbbsdhrfxquyayudxnkmru1md8tkc8µª½Ç2  requirements.txtqGü±W 6affo2jbZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64ô3´nY 6affo2jbZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64î®M*  6affo2jbb trainer/global_step2=$ž \  6affo2jbZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_644õßÇ/  6affo2jbb*"trainer/global_step2€¹/¹J  6affo2jbj9 ÛÝåÔð¹ó*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] )ø.­“  6affo2jbj  ÛÝåÔè¹Ô“î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 46.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ å12‘\ 6affo2jbjÊ ÛÝåÔˆÓÓ”·Trainable params: 48.8 M Non-trainable params: 86 Total params: 48.8 M Total estimated model params size (MB): 195.214 Modules in train mode: 5157 Modules in eval mode: 0 Total FLOPs: 0 Nœ·‚q 6affo2jbj` ÛÝåÔò”Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). qKIû{ 6affo2jbjj ÛÝåÔ¨úú”Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ZëH˜y 6affo2jbjh ÛÝåÔ舋•Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). î›Í€ 6affo2jbjo ÛÝåÔø©š•_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ·Æe(z 6affo2jbji ÛÝåÔð碕Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ë™õ7t 6affo2jbjc ÛÝåÔÀЯ•Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). PyQÒq 6affo2jbj` ÛÝåÔ𓾕Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ÿôžt 6affo2jbjc ÛÝåÔ˜¿Ç•Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). zb;n 6affo2jbj] ÛÝåÔÐÁΕMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ,´j†m 6affo2jbj\ ÛÝåÔè¤Õ•Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ¸M/xb 6affo2jb*Ð  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"¾¹Gîë 6affo2jbjÙ ÛÝåÔøýé›È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.  9"N 6affo2jbj¼ ÛÝåÔ¸ù›«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. A°R÷ê 6affo2jbjØ ÞÝåÔ ‘†EÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. õâO 6affo2jbj½ ÞÝåÔ Û—E­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Í’L;’‚(:ˆ éÝåÔ‘®ÉÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚36.0 gpu.0.powerWatts‚184.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã:f/‚(:¥ éÝåÔ¨™ÂÍ disk.drbd2266.out‚0) proc.memory.availableMB‚ 4805.65234375 disk.drbd2080.in‚0 disk.rbd1.in‚0" memory_percent‚70.668625831604 cpu‚25.019587303720613$ disk./.usageGB‚290.9688949584961 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 0.52734375 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 0.52734375 disk.dm-0.in‚0.078125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚ 52.8583984375 disk.drbd2080.out‚0 disk.rbd1.out‚0 network.recv‚41881 disk.sda3.out‚ 0.52734375" disk.nvme0n1.out‚ 52.8583984375 disk.sda1.in‚0 network.sent‚34512 disk.drbd2265.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚19.527166679828866 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2629.69921875* proc.memory.percent‚16.050410270690918$ disk.nvme0c0n1.out‚ 52.8583984375 disk.sda2.out‚0 disk.sda3.in‚0'•Ž’ ‚(:ˆ øÝåÔ²è¨Ç gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y èK(!‚(:ž øÝåÔ•þ¥É disk.dm-1.out‚1.015625 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 4804.74609375$ disk./.usageGB‚290.9744415283203 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.rbd1.out‚0.0625 disk.sda.in‚0! proc.memory.rssMB‚ 2629.734375 cpu‚25.02721396701028 disk.sda.out‚1.015625 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚1.015625 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 network.recv‚60196) disk./.usagePercent‚19.52753891478365 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 75.876953125 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚70.67415714263916 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.out‚ 75.876953125 disk.dm-0.out‚ 75.876953125 disk.drbd2266.out‚0) proc.memory.percent‚16.05062484741211 network.sent‚51628.=’"‚(:ˆ ‡ÞåÔÕ·¶Ç gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gž—G*#‚(:  ‡ÞåÔÈåÓÉ disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda.out‚ 1.5859375 disk.nvme0n1.in‚0.078125) proc.memory.percent‚16.05067253112793 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.out‚ 121.0390625$ memory_percent‚70.67549228668213" proc.memory.rssMB‚ 2629.7421875) disk./.usagePercent‚19.52805733004049 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0.0625 disk.sda.in‚0 disk.dm-0.in‚0.078125 disk.drbd1716.in‚0 network.sent‚64880 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 121.0390625 disk.sda2.out‚0 disk.dm-1.out‚ 1.5859375 cpu‚25.031769677023274$ disk./.usageGB‚290.9821662902832 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.out‚ 121.0390625 proc.cpu.threads‚17 network.recv‚77996 disk.sda3.in‚0) proc.memory.availableMB‚ 4804.52734375 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 1.58593753c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPtÆ0;-&) _wandbruntime‚46  _runtime‚46ó‡À'» & lr-AdamW/pg2‚6.243148256809893e-06 trainer/global_step‚49 % _timestamp‚1.7884403291801164e+09 _runtime‚ 46.227969546 _step‚0 & lr-AdamW/pg1‚6.243148256809893e-06NXÂ(½ & lr-AdamW/pg1‚6.243148256809893e-06 & lr-AdamW/pg2‚6.243148256809893e-06 trainer/global_step‚49 % _timestamp‚1.7884403291801164e+09 _runtime‚ 46.227969546 _step‚0Ä}èh)bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPî¨Äs*bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP¨ß-Dn+bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP¨¬£ºf,bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPTrmÔa-b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPaÃ>q.bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPEµ!èg/bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPÇFÚn0bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP§Ü„i1be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP[2¡jc2b_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP…˜øk8bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP¶žö¶p9bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPßÉýc:b_ bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP°uÌr?bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPï³Ožn@bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP—uíjAbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPó5óHsBbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPîòDorCbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP°ëÜ}pDbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP~$_Eb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP¿Fð@bFb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP×y‰`jGbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP¬‰ ÂsHbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPnc eoIbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPáu¿gJbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP&—!çqKbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP ‰¤ tLbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP¹ædMb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPX¡ý¶kNbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP„ñç8lObh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¹7vçhPbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP]-[]lQbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP!ó•FqRbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPNîÈoSbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP›ÚrßWTbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPKåÕ:hUbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP5¸”wpVbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPQó_lWbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPa]ÃmXbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP…;ä}tYbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÄÉû˜gZbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÖ ²Hh[bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP*·_i\be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP]î(e]ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPð€Aj^bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP<ä žr_bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPvæ>Gk`bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPð覊labh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP9ôç=qbbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPˆzÈ _cb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPŸä^xdbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP°ÃÆämebi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÇ~wsfbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPý™’†ngbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPS¥(1ihbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP¨ò´nibj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP9xÈþnjbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP6Uùokbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPÛ„ÚÎqlbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPrF^Êimbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÊˤâgnbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPU•eoba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP˜7_:gpbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPP¥9pqbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP:.ôXgrbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPŽ½úfsbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP>Œ^tbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP0_jaub] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPár™öjvbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPR&jkwbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¤Õ`xb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP¯ö¢Rvybr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPâx*×fzbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP„-­ºj{bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPV Æk|bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÀ‚ÅÐm}bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPê±Ip~bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP´@,Ælbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP8³¿y€bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPð^èfba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPàJ‘]o‚bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPMBÃíkƒbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPX›ƒ©o„bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPØ¡Ï0a…b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP²ím†bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP˜óÉ_‡bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP$špíoˆbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPº§¨®n‰bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP¨ t¬iŠbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP€É<˜p‹bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP`4¹gŒbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPv1a¡fba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPI~Î'gŽbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP AV8sbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP)2HIhbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP£º°t‘bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP>ȳˆn’bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP—zã$p“bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPªD\o”bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPVP«‹n•bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPØ„—c–b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPQeT3q—bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP€n§l˜bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPëW2qj™be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPˆàfÌgšbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPv&Ît›bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPl3"oœbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPu^–Gjbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPþ˜º^žbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPcèNchŸbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP+ÎÃlk bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPt5w¡br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP¢ ƒi¢bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP¸¹EÝl£bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPV3 ‰.¤) _wandbruntime‚46  _runtime‚46l{÷Ô×¥”O trainer/global_step‚49 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚861180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚763670 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 46828383232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3805721804800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1084122 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚508972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3406273550466 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26717 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45867641986 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1945304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207773184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3452528360960 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚861182 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚974 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3452545531392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020420096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3452545531392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020420096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012353024 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1062510 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3405475317890 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3359624715264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023309 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚254151 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3498574385664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3452528360960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1923690 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 49185954304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 46648835202 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1084122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1062510 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608yC‚ž'¦˜O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 46828383232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3805721804800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1084122 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚508972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3406273550466 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26717 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45867641986 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1945304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207773184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3452528360960 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚861182 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚974 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3452545531392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020420096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3452545531392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020420096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012353024 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1062510 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3405475317890 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3359624715264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023309 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚254151 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3498574385664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3452528360960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1923690 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 49185954304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 46648835202 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1084122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1062510 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 trainer/global_step‚49 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚861180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚763670~q:e§b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPÚ‘ý÷f¨ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPl¬ûße©b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPTéOšuªbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP$m(‘_«bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPNµäA^¬bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP!jÌéh­bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPtbéåo®bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÁløj¯be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPöý7ík°bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP„Óîn±bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPWÝ Rl²bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPÉ\³bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP§œîZb´b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPßA7 lµbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPÚ›k¶bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP2íEgi·bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP ÄØu¸bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP™ùe¹b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPäÎVmºbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP¯H}l»bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP@ý¸Žl¼bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPQÏñr½bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP$‘DÖq¾bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPß'mpl¿bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPkõsòeÀb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPôÃN1fÁba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP"ÚÖmÂbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP4ÒœáaÃb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP;uÌs^ÄbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPK i?gÅbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPr#Ú`Æb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPt5ëøfÇba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP¬¥éƒfÈba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPk*ÏbÉb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP=ÿiÊbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPdƒpËbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPhXDsÌbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPѶ9àjÍbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP!ˆ¡ÊqÎbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP(E]ÏbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÈDWIhÐbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPPŒ©mÑbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP-BmfÒba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPbd,{jÓbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPçÇ÷fÔba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP|Râ¥iÕbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPýXüÔkÖbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP;•³o×bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP5ûÄiØbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPl÷(dÙb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPmÚ*nÝbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP"­¸âoÞbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP‘OºÕwßbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPòèOônàbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPãvÔÁoábj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPן‰jâbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPdra&gãbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPwÔ6häbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPKu×råbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPk"¹iæbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPÛ˜èmçbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP‰©‚ykèbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPߥtrébm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPQdåëdêb_ n_òbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP>4¢Fmóbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPó;š.hôbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JP'µðiõbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPš©=föba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPJ­jig÷bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPÖCJ‰nøbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÝ i"jùbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPÜÿ‚Øgúbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP¬–Foûbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP甑mübh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPëyië`ýb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP$y÷µrþbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPåødpÿbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP®ß&e€b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPi¨~eb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP=Û8 k‚bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP®¹¦kƒbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPЙ''g„bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP죡Gm…bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP»·°_d†b_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1983704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203289600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3568932986880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3521177104896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚49 $ _timestamp‚1.788440329987377e+09 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3568932986880 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46973510144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3521159934464 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1083844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3521177104896 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚259698 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1962064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1983704 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1105482 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3931560604672 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 49953319936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚~ÜÚb518967 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1105482 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚878220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚557 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚778105 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228062208 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 47755881984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3881403995136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3426425464832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3568133444608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 47572791814 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚878222 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46973510144 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47755881984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3521159934464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3473217063222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207773184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚878220 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 47.03530423 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3521159934464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3473217063222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207773184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚878220 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 47.03530423 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1983704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203289600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3568932986880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3521177104896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚49 $ _timestamp‚1.788440329987377e+09 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3568932986880 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46973510144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3521159934464 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1083844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3521177104896 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚259698 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1962064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1983704 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1105482 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3931560604672 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 49953319936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚518967 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1105482 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚878220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚557 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚778105 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228062208 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 47755881984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3881403995136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3426425464832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3568133444608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 47572791814 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚878222 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46973510144 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47755881984 B loss/SRqqt4/assignment_loss"trainer/global_step2:JPmà°Ã@¬b; loss/FRt4/detection_loss"trainer/global_step2:JP©¹ßiB­b= loss/SRqqt1/detection_loss"trainer/global_step2:JPÆÚâ@®b; loss/FBt2/detection_loss"trainer/global_step2:JPÚ8lè7¯b2 loss/total_loss"trainer/global_step2:JPûÿMåA°b< loss/FBt1/assignment_loss"trainer/global_step2:JPÍçk¹A±b< loss/FBt3/assignment_loss"trainer/global_step2:JPfÖ0ÙA²b< loss/FBt4/assignment_loss"trainer/global_step2:JPçÑÚR@³b; loss/FRt3/detection_loss"trainer/global_step2:JPIÂG/@´b; loss/FBt3/detection_loss"trainer/global_step2:JPÏ„VÂAµb< loss/FRt3/assignment_loss"trainer/global_step2:JPU23&A¶b< loss/FBt2/assignment_loss"trainer/global_step2:JPËìFB·b= loss/SRqqt2/detection_loss"trainer/global_step2:JP³_Ä@¸b; loss/FRt2/detection_loss"trainer/global_step2:JP|˜*@¹b; loss/FBt4/detection_loss"trainer/global_step2:JPSªP8Cºb> loss/SRqqt3/assignment_loss"trainer/global_step2:JPÎÁsaB»b= loss/SRqqt4/detection_loss"trainer/global_step2:JPówG„C¼b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP¾öŒšC½b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP0èÇ8-¾b( epoch"trainer/global_step2:JPûqlu.¿) _wandbruntime‚47  _runtime‚47s³è'HÀ /loss/FRt1/assignment_loss‚7.152035971819344 /loss/FRt4/assignment_loss‚7.355044464732325 /loss/FRt1/detection_loss‚2.7886904947838542 0loss/SRqqt3/detection_loss‚2.774954143645508 .loss/FBt1/detection_loss‚6.591434070697198 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.120425188982928 1loss/SRqqt4/assignment_loss‚5.440623915195465 /loss/FRt4/detection_loss‚3.9925739390905512 1loss/SRqqt1/detection_loss‚2.4818381719864333 /loss/FBt2/detection_loss‚10.122820230967859 &loss/total_loss‚125.22896237838849 % _timestamp‚1.7884403299898772e+09 /loss/FBt1/assignment_loss‚2.762861499419579 /loss/FBt3/assignment_loss‚2.760997314710875 0loss/FBt4/assignment_loss‚2.7173096984624863 .loss/FRt3/detection_loss‚4.847997428739772 .loss/FBt3/detection_loss‚4.489956725288081 _runtime‚ 47.037008437 /loss/FRt3/assignment_loss‚7.351250536301556 0loss/FBt2/assignment_loss‚2.7488771466647877 0loss/SRqqt2/detection_loss‚4.195133775109198 _step‚3 .loss/FRt2/detection_loss‚6.225015166974454 /loss/FBt4/detection_loss‚10.946834415197372 1loss/SRqqt3/assignment_loss‚5.467108445508139 0loss/SRqqt4/detection_loss‚4.109876348823309 0loss/SRqqt1/assignment_loss‚5.37147483092088 1loss/SRqqt2/assignment_loss‚5.413828454366544 epoch‚0 {¸cLÁÆ /loss/FBt1/assignment_loss‚2.762861499419579 /loss/FBt3/assignment_loss‚2.760997314710875 0loss/FBt4/assignment_loss‚2.7173096984624863 .loss/FRt3/detection_loss‚4.847997428739772 .loss/FBt3/detection_loss‚4.489956725288081 _runtime‚ 47.037008437 /loss/FRt3/assignment_loss‚7.351250536301556 0loss/FBt2/assignment_loss‚2.7488771466647877 0loss/SRqqt2/detection_loss‚4.195133775109198 _step‚3 .loss/FRt2/detection_loss‚6.225015166974454 /loss/FBt4/detection_loss‚10.946834415197372 1loss/SRqqt3/assignment_loss‚5.467108445508139 0loss/SRqqt4/detection_loss‚4.109876348823309 0loss/SRqqt1/assignment_loss‚5.37147483092088 1loss/SRqqt2/assignment_loss‚5.413828454366544 epoch‚0 /loss/FRt1/assignment_loss‚7.152035971819344 /loss/FRt4/assignment_loss‚7.355044464732325 /loss/FRt1/detection_loss‚2.7886904947838542 0loss/SRqqt3/detection_loss‚2.774954143645508 .loss/FBt1/detection_loss‚6.591434070697198 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.120425188982928 1loss/SRqqt4/assignment_loss‚5.440623915195465 /loss/FRt4/detection_loss‚3.9925739390905512 1loss/SRqqt1/detection_loss‚2.4818381719864333 /loss/FBt2/detection_loss‚10.122820230967859 &loss/total_loss‚125.22896237838849 % _timestamp‚1.7884403299898772e+09å±þ¡“‚(:ˆ –ÞåÔï­ÈÇ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀêŒ-Â(:¢ –ÞåÔû̺Ê disk.sda.in‚0 disk.sda.out‚2.09375 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.sda3.out‚2.09375! disk.nvme0n1.out‚ 159.51171875 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.1884765625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 801.5078125 cpu‚25.03445732890455# disk.nvme0c0n1.out‚ 159.51171875 disk.drbd1716.in‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2629.7734375$ disk./.usageGB‚290.9898796081543 disk.dm-1.out‚2.09375$ memory_percent‚95.10798454284668) proc.memory.percent‚16.05086326599121 proc.cpu.threads‚17 network.sent‚90583* disk./.usagePercent‚19.528574977274722 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-0.out‚ 159.51171875 disk.rbd0.in‚0 network.recv‚110047 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.rbd0.out‚0×ÅÍ“Ä‚(:ˆ ¥ÞåÔÀòÊÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m^¯À=Å‚(:² ¥ÞåÔžŒßÊ( disk./.usagePercent‚19.5288222805528 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚95.10617256164551' proc.memory.availableMB‚ 801.8046875 disk.drbd2080.in‚0 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚200.63720703125 disk.sda.out‚ 2.66796875 disk.rbd0.in‚0 network.sent‚140670$ disk./.usageGB‚290.9935646057129 disk.dm-1.in‚0 disk.dm-1.out‚ 2.66796875 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚0.078125 proc.memory.rssMB‚ 2629.78125$ disk.nvme0n1.out‚200.63720703125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2.66796875) proc.memory.percent‚16.05091094970703 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚115801 disk.drbd1716.in‚0 disk.rbd1.out‚ 0.1884765625 disk.sda3.in‚0 cpu‚25.036617787255743 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚0.078125! disk.dm-0.out‚200.63720703125’Õ.Æ) _wandbruntime‚89  _runtime‚89~²4pÃǽ trainer/global_step‚99 % _timestamp‚1.7884403719956295e+09 _runtime‚ 89.042874048 _step‚4 ' lr-AdamW/pg1‚1.2613707702534273e-05 ' lr-AdamW/pg2‚1.2613707702534273e-05X¬úÇÈÁ _runtime‚ 89.042874048 _step‚4 ' lr-AdamW/pg1‚1.2613707702534273e-05 ' lr-AdamW/pg2‚1.2613707702534273e-05 trainer/global_step‚99 % _timestamp‚1.7884403719956295e+09Þ„t.É) _wandbruntime‚89  _runtime‚89ò‘PŒ'#ÊœO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6976541746688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3867356 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 97306790400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2132562 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93217069056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1713182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93217069056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1005 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 92860246610 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6976541746688 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 92434710528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 _step‚5 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6977341275648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1713180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194914304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2132562 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚505490 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1713182 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6884124206592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7589831321600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788440371996353e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6884107036160 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚506007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 €‰œÀ{QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 92079053394 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3845742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7589806548992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6884124206592 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 92434710528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700497122304Ø3™¦'Ë O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2132562 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚505490 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1713182 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6884124206592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7589831321600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788440371996353e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6884107036160 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚506007 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 92079053394 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3845742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7589806548992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6884124206592 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 92434710528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700497122304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6976541746688 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3867356 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 97306790400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2132562 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93217069056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1713182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93217069056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1005 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 92860246610 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6976541746688 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 92434710528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 _step‚5 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6977341275648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1713180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194914304a)U.Ì) _wandbruntime‚89  _runtime‚89w8ó/¬&ͦM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2153896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6952738609664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 98281499648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6767331950592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2175534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3905756 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1529086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1529610 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 trainer/global_step‚99 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6767314911232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884403727924492e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2153896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190706688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3905756 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1730220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3884116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7046900347904 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 93362195968 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1730222 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7665488739328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6952755780096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7046900347904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6952755780096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6861116153814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3905756 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1529086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1529610 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 trainer/global_step‚99 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6767314911232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884403727924492e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2153896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190706688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3905756 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1730220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3884116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7046900347904 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 93362195968 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1730222 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7665488739328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6952755780096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7046900347904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6952755780096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6861116153814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 Eÿ”:Ó‚(:¯ ´ÞåÔ¾«×É disk.rbd0.in‚0" proc.memory.rssMB‚ 2629.7890625 disk.drbd2265.out‚0 disk.sda2.in‚0 network.sent‚147859 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.sda.in‚0 cpu‚25.03666652031128 proc.cpu.threads‚17$ disk./.usageGB‚290.9955596923828 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚3.203125$ memory_percent‚95.11599540710449 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚3.203125* disk./.usagePercent‚19.528956172493206 disk.drbd1716.out‚0$ disk.nvme0n1.out‚237.69189453125 disk.dm-0.in‚0.078125 disk.sda1.in‚0' proc.memory.availableMB‚ 800.1953125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 0.5166015625 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚3.203125) proc.memory.percent‚16.05095863342285 network.recv‚121659! disk.dm-0.out‚237.69189453125 disk.drbd2266.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚237.69189453125 disk.drbd2266.out‚0 Þ¸Z“Ô‚(:ˆ ÃÞåÔ”ä¿Ç gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚189.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µØb¯9Õ‚(:® ÃÞåÔŒÑ®É disk.sda3.in‚0 disk.drbd1716.out‚0 disk.dm-0.out‚288.8974609375 disk.drbd2080.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2629.79296875 cpu‚25.038320238606136 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.5166015625 disk.sda2.in‚0 disk.dm-1.out‚ 3.8515625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.sent‚197666 disk.drbd1716.in‚0# disk.nvme0n1.out‚288.8974609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125$ memory_percent‚95.11985778808594 disk.sda.out‚ 3.8515625 disk.rbd0.in‚0 network.recv‚126809 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚288.8974609375$ proc.memory.availableMB‚799.5625 disk.sda1.in‚0* disk./.usagePercent‚19.529858855058936$ disk./.usageGB‚291.0090103149414 disk.sda3.out‚ 3.8515625 disk.dm-1.in‚0) proc.memory.percent‚16.05098247528076¨ü o“Ö‚(:ˆ ÒÞåÔƯßÈ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žR“ñ8ׂ(:­ ÒÞåÔÚçÊ disk.nvme0n1.in‚0.078125! proc.memory.rssMB‚ 2629.796875* proc.memory.percent‚16.051006317138672 disk.drbd2265.out‚0 disk.sda.out‚ 4.26171875 disk.sda3.out‚ 4.26171875 cpu‚25.039215942705294 disk.rbd1.in‚0 disk.rbd1.out‚ 0.6728515625 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125$ disk./.usageGB‚289.2037010192871 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 327.79296875* disk./.usagePercent‚19.408703033472186 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 327.79296875 disk.dm-1.out‚ 4.26171875 disk.drbd1716.in‚0! disk.nvme0n1.out‚ 327.79296875 network.sent‚204473 network.recv‚132066 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 799.19140625 disk.drbd2265.in‚0 disk.dm-1.in‚0# memory_percent‚95.1221227645874 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0ÛqÀ0Ø+ _wandbruntime‚131  _runtime‚131l ÔÔÅÙ¿ _runtime‚ 131.567544787 _step‚8 ' lr-AdamW/pg1‚1.8984267148258654e-05 ' lr-AdamW/pg2‚1.8984267148258654e-05 trainer/global_step‚149 % _timestamp‚1.7884404145203032e+09´†ÝTÉÚà _runtime‚ 131.567544787 _step‚8 ' lr-AdamW/pg1‚1.8984267148258654e-05 ' lr-AdamW/pg2‚1.8984267148258654e-05 trainer/global_step‚149 % _timestamp‚1.7884404145203032e+09ÐÆú{0Û+ _wandbruntime‚131  _runtime‚131wPº¼'ܶO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2565180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2565182 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3224226 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 146095836160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚ 131.568447932 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2565182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139605754880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10315685711360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2266076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚9 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚997 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10455308636672 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2565180 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138290464802 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 $ _timestamp‚1.788440414521034e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10179677033506 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10315702881792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1508175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10041386568704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3224226 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 145913504768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3202614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚149 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10454509107712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11373940838400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1017 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5767794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10315685711360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010ŽÃåþÀ'ݺO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11373940838400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1017 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5767794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10315685711360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2565180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2565182 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3224226 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 146095836160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚ 131.568447932 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2565182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139605754880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10315685711360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2266076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚9 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚997 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10455308636672 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2565180 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138290464802 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 $ _timestamp‚1.788440414521034e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10179677033506 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10315702881792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1508175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10041386568704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3224226 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 145913504768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3202614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚149 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10454509107712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 s1IV0Þ+ _wandbruntime‚132  _runtime‚132Ølæ˜ðßÁM QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3245586 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5806168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223867904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11449623028736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 139750881792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 $ _timestamp‚1.788440415317432e+09 _step‚10 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139750881792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10247418778838 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2582222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚149 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2582222 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1518167 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 _runtime‚ 132.365074998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10384317284864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11449598256128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10108204357632 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140533253632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140533253632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10248217011622 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2582220 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 146876806656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 199095296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3245586 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10108221396992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStaxÖY„×tsMonitor.on_train_batch_end/active_bytes.all.allocated‚10524867708928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚997 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11596475062784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5827808 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2582220 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10384334455296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 147075901952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3223948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5827808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280543 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139214421206 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚10104ÅËáË&àÅM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2582220 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 146876806656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 199095296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3245586 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10108221396992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10524867708928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚997 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11596475062784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5827808 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2582220 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10384334455296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 147075901952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3223948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5827808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280543 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139214421206 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3245586 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5806168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223867904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11449623028736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 139750881792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 $ _timestamp‚1.788440415317432e+09 _step‚10 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139750881792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10247418778838 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2582222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚149 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2582222 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1518167 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 _runtime‚ 132.365074998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10384317284864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11449598256128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10108204357632 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140533253632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140533253632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10248217011622 œ« 0á+ _wandbruntime‚132  _runtime‚132z/M7Hâ 0loss/FBt1/assignment_loss‚2.1004527333904717 -loss/FRt3/detection_loss‚4.47669098369385 .loss/FBt1/detection_loss‚1.774350855985711 1loss/SRqqt1/assignment_loss‚5.328492768203156 1loss/SRqqt3/assignment_loss‚5.550119774682181 1loss/SRqqt4/assignment_loss‚5.398047855922154 .loss/FRt2/detection_loss‚6.466723094383876 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.285070300102234 /loss/FRt3/assignment_loss‚7.446361014717503 .loss/FRt4/detection_loss‚5.647099774691366 /loss/FBt4/detection_loss‚29.342340389887493 _step‚11 /loss/FRt4/assignment_loss‚7.393662914153068 0loss/FBt3/assignment_loss‚2.1683249754064224 -loss/FBt4/assignment_loss‚2.4139430920283 0loss/SRqqt2/detection_loss‚4.602993870103681 0loss/SRqqt4/detection_loss‚4.737451933111463 /loss/FBt3/detection_loss‚10.474821285289877 &loss/total_loss‚145.15066916496795 /loss/FBt2/detection_loss‚11.196109356482824 epoch‚0 /loss/FRt2/assignment_loss‚7.235227108001709 1loss/SRqqt2/assignment_loss‚5.436569677816855 /loss/FBt2/assignment_loss‚2.359547662734985 /loss/FRt1/detection_loss‚2.7001795986667276 1loss/SRqqt1/detection_loss‚1.7899321046056627 $ _timestamp‚1.788440415319047e+09 0loss/SRqqt3/detection_loss‚1.826156040906374 _runtime‚ 132.366144927 *ô/LãÆ &loss/total_loss‚145.15066916496795 /loss/FBt2/detection_loss‚11.196109356482824 epoch‚0 /loss/FRt2/assignment_loss‚7.235227108001709 1loss/SRqqt2/assignment_loss‚5.436569677816855 /loss/FBt2/assignment_loss‚2.359547662734985 /loss/FRt1/detection_loss‚2.7001795986667276 1loss/SRqqt1/detection_loss‚1.7899321046056627 $ _timestamp‚1.788440415319047e+09 0loss/SRqqt3/detection_loss‚1.826156040906374 _runtime‚ 132.366144927 0loss/FBt1/assignment_loss‚2.1004527333904717 -loss/FRt3/detection_loss‚4.47669098369385 .loss/FBt1/detection_loss‚1.774350855985711 1loss/SRqqt1/assignment_loss‚5.328492768203156 1loss/SRqqt3/assignment_loss‚5.550119774682181 1loss/SRqqt4/assignment_loss‚5.398047855922154 .loss/FRt2/detection_loss‚6.466723094383876 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.285070300102234 /loss/FRt3/assignment_loss‚7.446361014717503 .loss/FRt4/detection_loss‚5.647099774691366 /loss/FBt4/detection_loss‚29.342340389887493 _step‚11 /loss/FRt4/assignment_loss‚7.393662914153068 0loss/FBt3/assignment_loss‚2.1683249754064224 -loss/FBt4/assignment_loss‚2.4139430920283 0loss/SRqqt2/detection_loss‚4.602993870103681 0loss/SRqqt4/detection_loss‚4.737451933111463 /loss/FBt3/detection_loss‚10.474821285289877 Ðwbb“ä‚(:ˆ áÞåÔ¼×ÄÇ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚189.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.9598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Á_…5å‚(:ª áÞåÔÝñÊ disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.051006317138672 network.sent‚210213% disk./.usageGB‚289.23205184936523 disk.drbd2265.in‚0 disk.rbd1.out‚ 0.6728515625 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2265.out‚0" disk.nvme0c0n1.out‚ 371.4765625$ memory_percent‚95.12524604797363! proc.memory.rssMB‚ 2629.796875 proc.cpu.threads‚17 disk.dm-1.out‚ 4.74609375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚135746) disk./.usagePercent‚19.41060568146666 disk.nvme0n1.out‚ 371.4765625 disk.sda.in‚0 disk.sda.out‚ 4.74609375 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 4.74609375 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚ 371.4765625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 798.6796875 cpu‚25.041838462342152«I’ žæ‚(:“ ðÞåÔŒŒ¬Ç gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚187.295' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.458999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚092ÿÏ0ç‚(:¥ ðÞåÔ—’§É disk.dm-1.out‚ 5.4296875 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 5.4296875 network.sent‚260423 network.recv‚141934 disk.drbd1716.in‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.051030158996582 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 409.609375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0n1.out‚ 409.609375 disk.drbd2080.out‚0 disk.sda.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚19.41190491970293$ disk./.usageGB‚289.2514114379883 disk.sda3.out‚ 5.4296875 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 cpu‚25.042741454627095 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.out‚ 409.609375# memory_percent‚95.1258897781372 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.7353515625( proc.memory.availableMB‚ 798.57421875# proc.memory.rssMB‚ 2629.80078125U¥’“è‚(:ˆ ÿÞåÔŒ®‘Ç gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.164' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0328 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fÿÿ7é‚(:¬ ÿÞåÔ­ã©Ê disk.dm-0.out‚ 450.44921875* proc.memory.percent‚16.051030158996582 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda.out‚ 6.06640625 disk.dm-1.out‚ 6.06640625 disk.drbd2265.in‚0' proc.memory.availableMB‚ 798.5390625# proc.memory.rssMB‚ 2629.80078125 disk.rbd0.out‚0 disk.sda3.out‚ 6.06640625 disk.dm-1.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚19.41279915402003 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 0.8212890625 disk.nvme0c0n1.in‚0.078125# disk.nvme0c0n1.out‚ 450.44921875 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0# memory_percent‚95.1261043548584 disk.sda1.in‚0 network.sent‚266812 network.recv‚146279 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 cpu‚25.04172503656638$ disk./.usageGB‚289.2647361755371 disk.rbd0.in‚0 disk.drbd1716.out‚0! disk.nvme0n1.out‚ 450.44921875âaŒ0ê+ _wandbruntime‚174  _runtime‚174ؽ]}ÆëÀ _runtime‚ 174.036099293 _step‚12 ' lr-AdamW/pg1‚2.5354826593983034e-05 ' lr-AdamW/pg2‚2.5354826593983034e-05 trainer/global_step‚199 % _timestamp‚1.7884404569889257e+09¼«8ÊìÄ ' lr-AdamW/pg2‚2.5354826593983034e-05 trainer/global_step‚199 % _timestamp‚1.7884404569889257e+09 _runtime‚ 174.036099293 _step‚12 ' lr-AdamW/pg1‚2.5354826593983034e-05 ˜W’Ð0í+ _wandbruntime‚174  _runtime‚174Lm@¾'î¸O OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3417180 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7689846 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 185212082176 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4294278 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13747264386560 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4272666 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15352617679360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7711460 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 184501876210 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3417182 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007772 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13932476468736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13747281556992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 194567324160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3417180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13747281556992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3018134 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1010359 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚199 $ _timestamp‚1.788440456989608e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13932476468736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13382276015104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚ 174.03703553 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4294278 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007775 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185994440704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1018 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15158025582592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 194376604160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚13 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1009835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13567576123890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 185212082176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3017607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 185994440704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13566777891314 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185283069426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 184501876210 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3417182 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007772 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13932476468736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13747281556992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes,#¶Ì­#.small_pool.peak‚ 1019218389 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 194567324160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3417180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13747281556992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3018134 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1010359 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚199 $ _timestamp‚1.788440456989608e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13932476468736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13382276015104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚ 174.03703553 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4294278 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007775 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185994440704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1018 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15158025582592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 194376604160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚13 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1009835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13567576123890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 185212082176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3017607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 185994440704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13566777891314 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185283069426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14002835069952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3434220 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7728220 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4315638 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7749860 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4315638 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1014791 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13634519636646 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14002035527680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7749860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186921939456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15233707772928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15233732545536 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 195347174400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4294000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13635317869430 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3033081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186207026038 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4294000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15429054947328 _step‚14 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3434222 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1015314 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 195544172544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3434222 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3032555 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2017764 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ª‹ÏCÎ&òÈM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚ 174.854918614 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13815913130496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13815895960064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196998144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14002835069952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 186139567616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14002835069952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3434220 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7728220 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4315638 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7749860 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4315638 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1014791 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13634519636646 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14002035527680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7749860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186921939456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15233707772928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15233732545536 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 195347174400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4294000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13635317869430 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3033081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186207026038 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4294000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15429054947328 _step‚14 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3434222 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1015314 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 195544172544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3434222 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3032555 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2017764 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 122599505928”ú 0ó+ _wandbruntime‚174  _runtime‚174Åw‡ÓNôÈ 0loss/FRt3/assignment_loss‚7.4790890799628365 1loss/SRqqt4/assignment_loss‚5.353317356109619 /loss/FBt2/detection_loss‚28.373083407680195 /loss/FBt4/detection_loss‚14.327111477653185 _step‚15 /loss/FRt2/assignment_loss‚7.134597846439907 0loss/SRqqt2/detection_loss‚3.447410963627757 0loss/SRqqt3/detection_loss‚1.664138842083048 &loss/total_loss‚139.14656657170798 1loss/SRqqt1/assignment_loss‚5.165650055056712 .loss/FRt4/detection_loss‚3.294303142556957 trainer/global_step‚199 0loss/FBt2/assignment_loss‚1.4387546678384144 /loss/FBt4/assignment_loss‚1.080553650856018 /loss/FBt1/detection_loss‚1.8677374381257887 .loss/FRt2/detection_loss‚5.125413578181045 0loss/FRt1/assignment_loss‚7.1439968651416255 /loss/FRt4/assignment_loss‚7.381294876921411 0loss/FBt1/assignment_loss‚0.7543726625113651 0loss/SRqqt4/detection_loss‚4.083874937146902 .loss/FBt3/detection_loss‚13.46921281172679 1loss/SRqqt3/assignment_loss‚5.402474838144639 .loss/FRt1/detection_loss‚3.164468535459509 /loss/FRt3/detection_loss‚3.5166217333740657 % _timestamp‚1.7884404578089447e+09 _runtime‚ 174.856091046 1loss/SRqqt2/assignment_loss‚5.229900866138692 0loss/FBt3/assignment_loss‚1.1712174186339745 0loss/SRqqt1/detection_loss‚2.077969520337513 epoch‚0gYf)RõÌ 0loss/FBt1/assignment_loss‚0.7543726625113651 0loss/SRqqt4/detection_loss‚4.083874937146902 .loss/FBt3/detection_loss‚13.46921281172679 1loss/SRqqt3/assignment_loss‚5.402474838144639 .loss/FRt1/detection_loss‚3.164468535459509 /loss/FRt3/detection_loss‚3.5166217333740657 % _timestamp‚1.7884404578089447e+09 _runtime‚ 174.856091046 1loss/SRqqt2/assignment_loss‚5.229900866138692 0loss/FBt3/assignment_loss‚1.1712174186339745 0loss/SRqqt1/detection_loss‚2.077969520337513 epoch‚0 0loss/FRt3/assignment_loss‚7.4790890799628365 1loss/SRqqt4/assignment_loss‚5.353317356109619 /loss/FBt2/detection_loss‚28.373083407680195 /loss/FBt4/detection_loss‚14.327111477653185 _step‚15 /loss/FRt2/assignment_loss‚7.134597846439907 0loss/SRqqt2/detection_loss‚3.447410963627757 0loss/SRqqt3/detection_loss‚1.664138842083048 &loss/total_loss‚139.14656657170798 1loss/SRqqt1/assignment_loss‚5.165650055056712 .loss/FRt4/detection_loss‚3.294303142556957 trainer/global_step‚199 0loss/FBt2/assignment_loss‚1.4387546678384144 /loss/FBt4/assignment_loss‚1.080553650856018 /loss/FBt1/detection_loss‚1.8677374381257887 .loss/FRt2/detection_loss‚5.125413578181045 0loss/FRt1/assignment_loss‚7.1439968651416255 /loss/FRt4/assignment_loss‚7.381294876921411Õ9ü§“ö‚(:ˆ ŽßåÔƒ×±Ç gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚190.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.1238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚>ìKé÷‚(:¢ ŽßåÔ¥™´É disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2629.80078125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 798.19921875 network.recv‚151634 disk.rbd0.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dmš$fmD-1.out‚6.65625 disk.rbd1.in‚0* disk./.usagePercent‚19.413315777224128 disk.nvme0c0n1.in‚0.078125 disk.dm-0.out‚ 494.015625 disk.drbd2265.in‚0 disk.rbd1.out‚ 0.9775390625! disk.nvme0c0n1.out‚ 494.015625 disk.sda.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.051030158996582 disk.sda3.in‚0 disk.sda3.out‚6.65625 network.sent‚273681% disk./.usageGB‚289.27243423461914 disk.sda.out‚6.65625 disk.nvme0n1.out‚ 494.015625$ memory_percent‚95.12817859649658 cpu‚25.042559062187962%ƒ“ø‚(:ˆ ßåÔ¾û¬Ç gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.483' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´³,#=ù‚(:² ßåÔé«É disk.drbd1716.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.9775390625 disk.sda1.in‚0 disk.drbd1716.in‚0! disk.dm-0.out‚535.39990234375 disk.drbd2080.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚19.413876689731648 disk.sda3.out‚7.171875 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125# proc.memory.rssMB‚ 2629.80078125 network.sent‚323101 network.recv‚156943 disk.sda.out‚7.171875 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0$ memory_percent‚95.12858390808105* proc.memory.percent‚16.051030158996582 disk.sda1.out‚0$ disk./.usageGB‚289.2807922363281 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0c0n1.out‚535.39990234375' proc.memory.availableMB‚ 798.1328125 disk.sda3.in‚0 disk.dm-1.out‚7.171875 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚535.39990234375 disk.rbd1.in‚0 cpu‚25.043133154828173 proc.cpu.threads‚17Wti’žú‚(:“ ¬ßåÔ«ìÑÇ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.834' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.766799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ýjCû‚(:¸ ¬ßåÔè„ÀÉ$ disk./.usageGB‚289.2889213562012 disk.dm-1.out‚ 7.66015625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 797.81640625 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 network.sent‚329578& disk.nvme0c0n1.out‚573.86474609375 disk.sda.in‚0 disk.sda3.out‚ 7.66015625 disk.drbd2080.out‚0 network.recv‚161767* disk./.usagePercent‚19.414422241787115 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 7.66015625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.out‚ 1.0400390625$ memory_percent‚95.13051509857178" proc.memory.rssMB‚ 2629.8046875 cpu‚25.043483579900002$ disk.nvme0n1.out‚573.86474609375! disk.dm-0.out‚573.86474609375 disk.rbd1.in‚0* proc.memory.percent‚16.051054000854492Ü”U©0ü+ _wandbruntime‚216  _runtime‚2168P‚õÆýÀ _step‚16 ' lr-AdamW/pg1‚3.1725386039707415e-05 ' lr-AdamW/pg2‚3.1725386039707415e-05 trainer/global_step‚249 % _timestamp‚1.7884404994408226e+09 _runtime‚ 216.488039458 ƒFÔÊþÄ ' lr-AdamW/pg2‚3.1725386039707415e-05 trainer/global_step‚249 % _timestamp‚1.7884404994408226e+09 _runtime‚ 216.488039458 _step‚16 ' lr-AdamW/pg1‚3.1725386039707415e-05s‰Þ0ÿ+ _wandbruntime‚216  _runtime‚216Èõ¶¿'€¹O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4269180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 231600768000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2507372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1263616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18942135099392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 trainer/global_step‚249 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232383126528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5342718 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 $ _timestamp‚1.788440499441494e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16723182500864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17410443829760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 _step‚17 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 242908969984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230713287618 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 231600768000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5364330 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18942159872000 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5364330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 231494480834 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9611898 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3770988 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19185251173376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16723165461504 _runtime‚ 216.488958452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17178843061760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2507375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1264134 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16953878749122 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4269182 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4269180 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17411243358720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5342718 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17178860232192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16954676981698 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3771509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17178843061760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9633512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0?wóÒÃ'½O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18942135099392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 trainer/global_step‚249 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232383126528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5342718 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 $ _timestamp‚1.788440499441494e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16723182500864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17410443829760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 _step‚17 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 242908969984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230713287618 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 231600768000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5364330 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18942159872000 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5364330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 231494480834 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9611898 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3770988 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19185251173376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16723165461504 _runtime‚ 216.488958452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17178843061760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2507375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1264134 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16953878749122 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4269182 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4269180 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17411243358720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5342718 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17178860232192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16954676981698 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3771509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17178843061760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9633512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4269180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 231600768000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2507372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1263616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317_ŠÌÀ0‚+ _wandbruntime‚217  _runtime‚217 áR† ƒÄM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17247491805696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2517367 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5385690 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17480802430976 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5364052 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19017842062336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5364052 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5385690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 232528253440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19261912270336 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231637244022 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3786609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9671912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4286220 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 232528253440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196998144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9650272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 217.285184923 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2517364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 243873209856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9650272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19261690499584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17021620494454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16790000289792 % _timestamp‚1.7884405002374032e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232418437446 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.currehš¿9¼nt‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4286220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3786078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17480802430976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 trainer/global_step‚249 _step‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17247474635264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272‘lUÎ&„ÈM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17247474635264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17247491805696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2517367 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5385690 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17480802430976 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5364052 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19017842062336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5364052 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5385690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 232528253440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19261912270336 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231637244022 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3786609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9671912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4286220 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 232528253440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196998144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9650272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 217.285184923 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2517364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 243873209856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9650272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19261690499584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17021620494454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16790000289792 % _timestamp‚1.7884405002374032e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232418437446 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4286220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3786078 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17480802430976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 trainer/global_step‚249 _step‚18%¼Ò0…+ _wandbruntime‚217  _runtime‚217jDê˜M†Ç 1loss/SRqqt2/assignment_loss‚4.045761938278492 .loss/FRt4/detection_loss‚3.832602359678434 _runtime‚ 217.286189649 .loss/FRt2/assignment_loss‚7.12952396273613 0loss/FBt1/assignment_loss‚0.3853459379663973 1loss/SRqqt1/detection_loss‚1.8231542754735126 /loss/FBt2/detection_loss‚15.305475866252726 .loss/FRt1/detection_loss‚2.731864611485175 .loss/FRt2/detection_loss‚6.485561065375805 /loss/FRt3/detection_loss‚2.8926019598290607 0loss/SRqqt4/detection_loss‚5.770030824201448 /loss/FRt3/assignment_loss‚7.444976401778887 0loss/FBt3/assignment_loss‚0.5837315342256001 1loss/SRqqt3/detection_loss‚1.8570392522109613 .loss/FBt4/detection_loss‚6.691223422288894 $ _timestamp‚1.788440500239048e+09 .loss/FRt1/assignment_loss‚7.19129467861993 0loss/FBt2/assignment_loss‚0.5258157730779864 0loss/FBt4/assignment_loss‚0.5552856934070587 &loss/total_loss‚107.23304285712268 _step‚19 /loss/FRt4/assignment_loss‚7.307908161826756 1loss/SRqqt1/assignment_loss‚3.788642085966517 1loss/SRqqt3/assignment_loss‚4.260809421539307 1loss/SRqqt2/detection_loss‚3.2894116001060376 /loss/FBt1/detection_loss‚1.0264785108764882 .loss/FBt3/detection_loss‚8.133859952290853 2loss/SRqqt4/assignment_loss‚4.1746435676302225 epoch‚0 trainer/global_step‚249¾ÒÄUQ‡Ë .loss/FRt1/assignment_loss‚7.19129467861993 0loss/FBt2/assignment_loss‚0.5258157730779864 0loss/FBt4/assignment_loss‚0.5552856934070587 &loss/total_loss‚107.23304285712268 _step‚19 /loss/FRt4/assignment_loss‚7.307908161826756 1loss/SRqqt1/assignment_loss‚3.788642085966517 1loss/SRqqt3/assignment_loss‚4.260809421539307 1loss/SRqqt2/detection_loss‚3.2894116001060376 /loss/FBt1/detection_loss‚1.0264785108764882 .loss/FBt3/detection_loss‚8.133859952290853 2loss/SRqqt4/assignment_loss‚4.1746435676302225 epoch‚0 trainer/global_step‚249 1loss/SRqqt2/assignment_loss‚4.045761938278492 .loss/FRt4/detection_loss‚3.832602359678434 _runtime‚ 217.286189649 .loss/FRt2/assignment_loss‚7.12952396273613 0loss/FBt1/assignment_loss‚0.3853459379663973 1loss/SRqqt1/detection_loss‚1.8231542754735126 /loss/FBt2/detection_loss‚15.305475866252726 .loss/FRt1/detection_loss‚2.731864611485175 .loss/FRt2/detection_loss‚6.485561065375805 /loss/FRt3/detection_loss‚2.8926019598290607 0loss/SRqqt4/detection_loss‚5.770030824201448 /loss/FRt3/assignment_loss‚7.444976401778887 0loss/FBt3/assignment_loss‚0.5837315342256001 1loss/SRqqt3/detection_loss‚1.8570392522109613 .loss/FBt4/detection_loss‚6.691223422288894 $ _timestamp‚1.788440500239048e+09!?Žv“ˆ‚(:ˆ »ßåÔò€¼Ç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.369' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«í÷:‰‚(:¯ »ßåÔÌóÈÉ network.recv‚166295 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.078125% proc.memory.availableMB‚ 797.71875 cpu‚25.04485385174575 disk.sda2.in‚0 network.sent‚336030% disk./.usageGB‚289.30200576782227 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚616.20654296875 disk.drbd2266.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.051054000854492 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0$ memory_percent‚95.13111114501953 proc.cpu.threads‚17! disk.dm-0.out‚616.20654296875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2629.8046875 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚616.20654296875 disk.sda.out‚8.140625 disk.dm-1.out‚8.140625 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.0400390625 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚8.140625 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.sda3.in‚0* disk./.usagePercent‚19.415300347629557 disk.sda1.in‚0¡1цžŠ‚(:“ ÊßåÔð¿Ç gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.123' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.624599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¼§…9‹‚(:® ÊßåÔ‚¸ÒÉ" proc.memory.rssMB‚ 2629.8046875* proc.memory.percent‚16.051054000854492 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.04419195153879 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚652.97216796875 disk.sda3.out‚ 8.6484375 disk.dm-1.out‚ 8.6484375! disk.dm-0.out‚652.97216796875 disk.rbd1.out‚ 1.166015625 disk.sda.in‚0) disk./.usagePercent‚19.41593499030694% disk./.usageGB‚289.31146240234375 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.availableMB‚797.875 network.recv‚172437$ memory_percent‚95.13015747070312 disk.sda1.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚652.97216796875 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 8.6484375 disk.nvme0c0n1.in‚0.078125 network.sent‚386429 disk.drbd2265.in‚0 disk.sda2.out‚0³ž’Œ‚(:‡ ÙßåÔŸßœÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.027' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÒjlA‚(:¶ ÙßåÔºÄŸÊ disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0$ memory_percent‚95.12555599212646 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 9.22265625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 cpu‚25.04350032534624& disk.nvme0c0n1.out‚688.25244140625 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 9.22265625 disk.dm-1.out‚ 9.22265625$ disk.nvme0n1.out‚688.25244140625 disk.rbd1.out‚ 1.166015625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125! disk.dm-0.out‚688.25244140625 network.sent‚392375 network.recv‚176500% disk./.usageGB‚289.30889892578125( proc.memory.availableMB‚ 798.62890625* proc.memory.percent‚16.051054000854492 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2629.8046875 disk.dm-0.in‚0.078125) disk./.usagePercent‚19.41576295324393"SÒº0Ž+ _wandbruntime‚258  _runtime‚258ŸÝ?Ùľ $ _timestamp‚1.788440541817756e+09 _runtime‚ 258.86497033 _step‚20 ' lr-AdamW/pg1‚3.8095945485431796e-05 ' lr-AdamW/pg2‚3.8095945485431796e-05 trainer/global_step‚299®oÈ ' lr-AdamW/pg1‚3.8095945485431796e-05 ' lr-AdamW/pg2‚3.8095945485431796e-05 trainer/global_step‚299 $ _timestamp‚1.788440541817756e+09 _runtime‚ 258.86497033 _step‚20ç›Ã£0‘+ _wandbruntime‚258  _runtime‚258`-ŒÐÄ'’¾O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20610438907392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20064054907904 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 276924699026 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20610421736960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277989453824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20610421736960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3006975 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3006972 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20340979606930 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20610438907392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20064071947264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278771812352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884405418184786e+09 _runtime‚ 258.865921104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11555564 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20888411190784 trainer/global_step‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5121182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4522758 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 291451143680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1018 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278771812352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1516344 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 291648155136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚21Ž}  “ÂO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5121180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20610421736960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3006975 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3006972 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20340979606930 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20610438907392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20064071947264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278771812352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884405418184786e+09 _runtime‚ 258.865921104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11555564 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20888411190784 trainer/global_step‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5121182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4522758 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 291451143680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1018 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278771812352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1516344 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 291648155136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚21 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20610438907392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20064054907904 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 276924699026 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20610421736960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277989453824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5138220 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6434104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4538332 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4537786 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278916939264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1521365 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1520822 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 292425990144 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6434104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 _step‚22 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20130889736192 _runtime‚ 259.663757968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5138222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23094352796672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚299 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20958769792000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 292618793984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20679070480896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20409519585046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11593964 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20958769792000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22801926806528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11572324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20679053310464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 277848655430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11572324 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6455742 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11593964 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389xêcÒ&–ÌM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20130889736192 _runtime‚ 259.663757968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5138222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23094352796672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚299 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20958769792000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 292618793984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20679070480896 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20409519585046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11593964 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20958769792000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22801926806528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11572324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20679053310464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 277848655430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11572324 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6455742 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11593964 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23094570373120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5138220 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6434104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4538332 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4537786 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278916939264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1521365 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1520822 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 292425990144 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6434104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 _step‚22 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚5436A¯0—+ _wandbruntime‚259  _runtime‚259ø¿¬M˜Ç .loss/FRt1/detection_loss‚3.029151743421188 .loss/FRt3/detection_loss‚4.150020751729608 /loss/FBt1/detection_loss‚1.5084293509729094 %loss/total_loss‚99.40989575681007 /loss/FRt2/assignment_loss‚7.245867404070768 1loss/SRqqt4/assignment_loss‚3.573686038454374 .loss/FRt4/detection_loss‚3.366241669952869 0loss/SRqqt3/detection_loss‚1.872285360828934 /loss/FBt2/detection_loss‚16.521239751577376 0loss/FBt4/assignment_loss‚0.3603161473170315 0loss/SRqqt1/detection_loss‚2.114032213666266 trainer/global_step‚299 .loss/FRt2/detection_loss‚5.156876284967769 0loss/FRt1/assignment_loss‚7.1693710180429315 2loss/SRqqt1/assignment_loss‚3.3578176729140745 2loss/SRqqt2/assignment_loss‚3.5144160403761755 0loss/SRqqt2/detection_loss‚3.860519056749898 /loss/SRqqt4/detection_loss‚3.33454325205336 2loss/SRqqt3/assignment_loss‚3.9165835868228567 % _timestamp‚1.7884405426175704e+09 0loss/FBt3/assignment_loss‚0.3429356120865453 /loss/FRt4/assignment_loss‚7.433247108459472 0loss/FBt1/assignment_loss‚0.6144111895259341 0loss/FBt2/assignment_loss‚0.5491978721693158 .loss/FBt3/detection_loss‚4.922189898563154 .loss/FBt4/detection_loss‚4.150877646317608 epoch‚0 _runtime‚ 259.664724041 _step‚23 /loss/FRt3/assignment_loss‚7.3456390857696536lèQ™Ë 0loss/SRqqt3/detection_loss‚1.872285360828934 /loss/FBt2/detection_loss‚16.521239751577376 0loss/FBt4/assignment_loss‚0.3603161473170315 0loss/SRqqt1/detection_loss‚2.114032213666266 trainer/global_step‚299 .loss/FRt2/detection_loss‚5.156876284967769 0loss/FRt1/assignment_loss‚7.1693710180429315 2loss/SRqqt1/assignment_loss‚3.3578176729140745 2loss/SRqqt2/assignment_loss‚3.5144160403761755 0loss/SRqqt2/detection_loss‚3.860519056749898 /loss/SRqqt4/detection_loss‚3.33454325205336 2loss/SRqqt3/assignment_loss‚3.9165835868228567 % _timestamp‚1.7884405426175704e+09 0loss/FBt3/assignment_loss‚0.3429356120865453 /loss/FRt4/assignment_loss‚7.433247108459472 0loss/FBt1/assignment_loss‚0.6144111895259341 0loss/FBt2/assignment_loss‚0.5491978721693158 .loss/FBt3/detection_loss‚4.922189898563154 .loss/FBt4/detection_loss‚4.150877646317608 epoch‚0 _runtime‚ 259.664724041 _step‚23 /loss/FRt3/assignment_loss‚7.345639085769653 .loss/FRt1/detection_loss‚3.029151743421188 .loss/FRt3/detection_loss‚4.150020751729608 /loss/FBt1/detection_loss‚1.5084293509729094 %loss/total_loss‚99.40989575681007 /loss/FRt2/assignment_loss‚7.245867404070768 1loss/SRqqt4/assignment_loss‚3.573686038454374 .loss/FRt4/detection_loss‚3.366241669952869²,)“š‚(:ˆ èßåÔÚ’¿Ç gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ËÖ;›‚(:° èßåÔ”üÊÊ disk.sda1.in‚0 disk.sda.in‚0$ disk./.usageGB‚289.3201370239258 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.078125$ memory_percent‚95.12474536895752 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 network.sent‚398669 disk.dm-1.out‚ 9.58203125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚25.04461567710766 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.out‚ 732.365234375 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 732.365234375( proc.memory.availableMB‚ 798.76171875 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚ 732.365234375 disk.rbd0.in‚0 disk.sda.out‚ 9.58203125* proc.memory.percent‚16.051054000854492 network.recv‚181066 disk.sda3.out‚ 9.58203125 disk.sda2.out‚0" proc.memory.rssMB‚ 2629.8046875* disk./.usagePercent‚19.416517151439802 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.228515625À”‚“œ‚(:ˆ ÷ßåÔ’ΊÇ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.307' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o°yö‚(:® ÷ßåÔòÚ–Ê disk.rbd0.out‚0 disk.drbd2266.in‚0 network.sent‚448661) disk./.usagePercent‚19.41638172345419 disk.rbd1.out‚ 1.228515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚10.21875 disk.sda3.in‚0 disk.sda3.out‚10.21875 disk.sda.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚95.12488842010498 disk.rbd0.in‚0 disk.dm-0.out‚786.2353515625 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2629.8046875 disk.nvme0c0n1.in‚0.078125 disk.dm-1.out‚10.21875# disk.nvme0n1.out‚786.2353515625 disk.drbd2266.out‚0 disk.dm-0.in‚0.078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.051054000854492 network.recv‚186588% disk./.usageGB‚289.31811904907227% disk.nvme0c0n1.out‚786.2353515625 disk.drbd1716.out‚0( proc.memory.availableMB‚ 798.73828125 cpu‚25.044139209626067 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 üÈàCdisk.drbd2265.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0¶Âcžž‚(:“ †àåÔÁšåÇ gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.593' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.718599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚020w^8Ÿ‚(:­ †àåÔ©šÓÉ network.recv‚191649$ disk./.usageGB‚289.3611488342285 disk.dm-1.out‚10.71875 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.3544921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2629.8046875 network.sent‚455292 disk.dm-1.in‚0# disk.nvme0n1.out‚832.0517578125 disk.sda1.in‚0 disk.sda.out‚10.71875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚832.0517578125) disk./.usagePercent‚19.41926948844042 disk.rbd1.in‚0$ memory_percent‚95.12536525726318 disk.sda3.out‚10.71875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0( proc.memory.availableMB‚ 798.66015625 disk.drbd2080.in‚0 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚832.0517578125* proc.memory.percent‚16.051054000854492 disk.drbd1716.in‚0 disk.sda1.out‚0 cpu‚25.044492741445327ƒ®Æ0 + _wandbruntime‚301  _runtime‚301è—u-Æ¡À ' lr-AdamW/pg1‚4.4466504931156176e-05 ' lr-AdamW/pg2‚4.4466504931156176e-05 trainer/global_step‚349 % _timestamp‚1.7884405844093666e+09 _runtime‚ 301.456651146 _step‚24ïƳ梀 ' lr-AdamW/pg1‚4.4466504931156176e-05 ' lr-AdamW/pg2‚4.4466504931156176e-05 trainer/global_step‚349 % _timestamp‚1.7884405844093666e+09 _runtime‚ 301.456651146 _step‚24˜ž%}0£+ _wandbruntime‚301  _runtime‚301»ìÜÒÄ'¤¾O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 _runtime‚ 301.457505589 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367178080768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5973180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 _step‚25 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24366378551808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5275941 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23728080464738 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24042017582592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5973182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26850722719744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7482822 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 325160498176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884405844100597e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5973180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7504434 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 324378139648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24366378551808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 323917303650 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3506572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1769910 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367178080768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5973180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 _step‚25 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24366378551808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5275941 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23728080464738 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24042017582592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5973182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26850722719744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7482822 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 325160498176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884405844100597e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5973180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7504434 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 324378139648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24366378551808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 323917303650 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3506572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1769910;i–P0¦+ _wandbruntime‚302  _runtime‚302‡ähÎ&§ÈM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23471779182592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26586036323328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23795822210070 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26927373472256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23796620442854 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 324841260262 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24435937610752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5990220 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7525794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26927160090112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23471762143232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5291637 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24436737153024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24110649156096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26586061095936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3516567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7525794 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24110631985664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884405852172513e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13494376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1775070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚ 302.265105207 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5990222 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24436737153024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5990220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 325305625088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 326087996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5291094 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 325305625088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13516016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 326087996928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3516564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13516016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13494376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 341123766784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 324060066838 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24110649156096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5990220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 325305625088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 326087996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5291094 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 325305625088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13516016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 326087996928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3516564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13516016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13494376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 341123766784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 324060066838 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24110649156096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23471779182592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26586036323328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23795822210070 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26927373472256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23796620442854 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 324841260262 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24435937610752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5990220 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7525794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26927160090112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23471762143232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5291637 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24436737153024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24110649156096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26586061095936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3516567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7525794 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24110631985664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884405852172513e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424ë¢h0©+ _wandbruntime‚302  _runtime‚302äã¯LªÆ 1loss/SRqqt1/detection_loss‚2.5460160829579266 % _timestamp‚1.7884405852190218e+09 _runtime‚ 302.266149287 0loss/SRqqt2/assignment_loss‚3.38934414936946 2loss/SRqqt3/assignment_loss‚3.8637860298156737 .loss/FRt2/detection_loss‚4.808962763774963 2loss/SRqqt4/assignment_loss‚3.6199134560518487 0loss/FBt1/assignment_loss‚0.6526791825890541 /loss/FBt4/assignment_loss‚0.613509491700679 .loss/FBt1/detection_loss‚7.872424363201705 .loss/FBt2/detection_loss‚23.41812485030719 .loss/FBt3/detection_loss‚2.442238479024834 -loss/FRt1/assignment_loss‚7.2057004255407 1loss/SRqqt1/assignment_loss‚3.262841818549416 /loss/FRt1/detection_loss‚3.1016865619549563 .loss/FRt3/detection_loss‚4.328778912362299 -loss/FBt4/detection_loss‚6.55914053440094 /loss/FRt3/assignment_loss‚7.406502585662039 .loss/FRt4/detection_loss‚3.688771151834064 epoch‚0 _step‚27 0loss/FRt4/assignment_loss‚7.3237981372409395 1loss/SRqqt2/detection_loss‚2.4678587118020423 1loss/SRqqt4/detection_loss‚3.6781078497337742 /loss/FRt2/assignment_loss‚7.270240965343657 0loss/FBt2/assignment_loss‚0.6374679827796561 1loss/FBt3/assignment_loss‚0.24165900093417061 trainer/global_step‚349 0loss/SRqqt3/detection_loss‚1.609937398763941 &loss/total_loss‚112.00949088569593©‰P«Ê _step‚27 0loss/FRt4/assignment_loss‚7.3237981372409395 1loss/SRqqt2/detection_loss‚2.4678587118020423 1loss/SRqqt4/detection_loss‚3.6781078497337742 /loss/FRt2/assignment_loss‚7.270240965343657 0loss/FBt2/assignment_loss‚0.6374679827796561 1loss/FBt3/assignment_loss‚0.24165900093417061 trainer/global_step‚349 0loss/SRqqt3/detection_loss‚1.609937398763941 &loss/total_loss‚112.00949088569593 1loss/SRqqt1/detection_loss‚2.5460160829579266 % _timestamp‚1.7884405852190218e+09 _runtime‚ 302.266149287 0loss/SRqqt2/assignment_loss‚3.38934414936946 2loss/SRqqt3/assignment_loss‚3.8637860298156737 .loss/FRt2/detection_loss‚4.808962763774963 2loss/SRqqt4/assignment_loss‚3.6199134560518487 0loss/FBt1/assignment_loss‚0.6526791825890541 /loss/FBt4/assignment_loss‚0.613509491700679 .loss/FBt1/detection_loss‚7.872424363201705 .loss/FBt2/detection_loss‚23.41812485030719 .loss/FBt3/detection_loss‚2.442238479024834 -loss/FRt1/assignment_loss‚7.2057004255407 1loss/SRqqt1/assignment_loss‚3.262841818549416 /loss/FRt1/detection_loss‚3.1016865619549563 .loss/FRt3/detection_loss‚4.328778912362299 -loss/FBt4/detection_loss‚6.55914053440094 /loss/FRt3/assignment_loss‚7.406502585662039 .loss/FRt4/detection_loss‚3.688771151834064 epoch‚0 ŸPœ¬‚(:‘ •àåÔ†Ñ¯Ç gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.55' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚38.11000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…׋ýB­‚(:· •àåÔ”ßµÉ disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.out‚ 1.3544921875 disk.sda1.in‚0 network.sent‚460900 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚95.12579441070557 network.recv‚195296 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚887.4091796875 disk.dm-1.out‚ 11.15234375 disk.dm-0.out‚887.4091796875 disk.drbd2266.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2629.8046875 disk.sda.out‚ 11.15234375 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 cpu‚25.04403633932506 disk.drbd1716.in‚0) disk./.usagePercent‚19.42090921669723 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 11.15234375 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚289.38558197021484# disk.nvme0n1.out‚887.4091796875( proc.memory.availableMB‚ 798.58984375* proc.memory.percent‚16.0510540008544923zBi“®‚(:ˆ ¤àåÔåÿ¹Ç gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.862' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a±ï»?¯‚(:´ ¤àåÔž…¿É disk.sda.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚16.051077842712402 proc.cpu.threads‚17 disk.dm-1.out‚ 11.5546875 cpu‚25.04497868998846$ disk./.usageGB‚289.3973922729492 disk.sda3.out‚ 11.5546875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 network.sent‚510919 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 11.5546875 disk.drbd1716.out‚0 disk.drbd2266.in‚0$ memory_percent‚95.12467384338379# proc.memory.rssMB‚ 2629.80859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0* disk./.usagePercent‚19.421701816023237 disk.nvme0n1.in‚0.078125# disk.nvme0n1.out‚930.1669921875 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.4169921875' proc.memory.availableMB‚ 798.7734375 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.dm-0.out‚930.1669921875 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚930.1669921875 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚201263J}êƒ0°+ _wandbruntime‚344  _runtime‚344öã>ñ½ _runtime‚ 344.044969605 _step‚28 & lr-AdamW/pg1‚5.083706437688056e-05 & lr-AdamW/pg2‚5.083706437688056e-05 trainer/global_step‚399 $ _timestamp‚1.788440626997777e+099Å}ÏDzÁ & lr-AdamW/pg1‚5.083706437688056e-05 & lr-AdamW/pg2‚5.083706437688056e-05 trainer/global_step‚399 $ _timestamp‚1.788440626997777e+09 _runtime‚ 344.044969605 _step‚28å`a80³+ _wandbruntime‚344  _runtime‚344 "g_Ä'´¾O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30294488422400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8552874 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26745833800704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15378054 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30294463649792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27115181322546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884406269984348e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27844345912832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27473579087360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27473596257792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27115979555122 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6825180 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6825180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27473596257792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27845145441792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521ö¯»µÂO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370766825472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30683661419520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26745850840064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 371549184000 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6029521 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 LBDeviceStatsMonitor.on_train_batch_start/alèÁ™& location.small_pool.peak‚26470 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6825180 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6825180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27473596257792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27845145441792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30294488422400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8552874 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26745833800704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15378054 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30294463649792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27115181322546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884406269984348e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27844345912832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27473579087360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27473596257792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27115979555122 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6044615 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8574208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30370145840128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6842222 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372476682752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6842220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27183721300662 _step‚30 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15438068 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1018 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884406278032405e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184415232 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30760324855296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30370170612736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8595846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 371694310912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26812651589632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27542227831296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚399 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15416428 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27542210660864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15416428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27914704514048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6045144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 344.850851295 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328Iÿ~HÒ&¸ÌM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2028451 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 370271478246 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6044615 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8574208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30370145840128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6842222 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372476682752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6842220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27183721300662 _step‚30 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15438068 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1018 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884406278032405e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184415232 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30760324855296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30370170612736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8595846 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 371694310912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26812651589632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27542227831296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚399 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15416428 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27542210660864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15416428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27914704514048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6045144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 344.850851295 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328…!¹0¹+ _wandbruntime‚344  _runtime‚344ªÿ³xOºÉ epoch‚0 % _timestamp‚1.7884406278048227e+09 0loss/FBt1/assignment_loss‚0.7392261713713167 _runtime‚ 344.851922677 /loss/FRt3/assignment_loss‚7.427640175819397 /loss/SRqqt4/assignment_loss‚3.5215370288262 .loss/FRt2/detection_loss‚4.421579944816503 1loss/SRqqt2/detection_loss‚2.2360617520682737 .loss/FRt3/detection_loss‚4.045566190034151 /loss/SRqqt4/detection_loss‚4.17136268890821 /loss/FBt1/detection_loss‚1.9976235870156789 .loss/FBt4/detection_loss‚4.875809199176729 trainer/global_step‚399 _step‚31 /loss/FRt1/assignment_loss‚7.233635578515394 0loss/FBt2/assignment_loss‚0.7851797821000218 2loss/SRqqt1/assignment_loss‚3.4031551281611123 2loss/SRqqt2/assignment_loss‚3.3525902863704795 0loss/FBt4/assignment_loss‚0.3297103027725825 .loss/FRt1/detection_loss‚3.195270121413863 %loss/total_loss‚98.24796016637694 1loss/FBt3/assignment_loss‚0.27420370863416255 /loss/FRt4/detection_loss‚3.4782896073574716 1loss/SRqqt1/detection_loss‚1.5668172647035843 0loss/FRt2/assignment_loss‚7.1581667770038955 /loss/FRt4/assignment_loss‚7.333701620710657 1loss/SRqqt3/assignment_loss‚3.817729688459827 1loss/SRqqt3/detection_loss‚2.7121976255373186 .loss/FBt2/detection_loss‚16.14710991382599 /loss/FBt3/detection_loss‚4.0237960227741265+’IêS»Í .loss/FRt3/detection_loss‚4.045566190034151 /loss/SRqqt4/detection_loss‚4.17136268890821 /loss/FBt1/detection_loss‚1.9976235870156789 .loss/FBt4/detection_loss‚4.875809199176729 trainer/global_step‚399 _step‚31 /loss/FRt1/assignment_loss‚7.233635578515394 0loss/FBt2/assignment_loss‚0.7851797821000218 2loss/SRqqt1/assignment_loss‚3.4031551281611123 2loss/SRqqt2/assignment_loss‚3.3525902863704795 0loss/FBt4/assignment_loss‚0.3297103027725825 .loss/FRt1/detection_loss‚3.195270121413863 %loss/total_loss‚98.24796016637694 1loss/FBt3/assignment_loss‚0.27420370863416255 /loss/FRt4/detection_loss‚3.4782896073574716 1loss/SRqqt1/detection_loss‚1.5668172647035843 0loss/FRt2/assignment_loss‚7.1581667770038955 /loss/FRt4/assignment_loss‚7.333701620710657 1loss/SRqqt3/assignment_loss‚3.817729688459827 1loss/SRqqt3/detection_loss‚2.7121976255373186 .loss/FBt2/detection_loss‚16.14710991382599 /loss/FBt3/detection_loss‚4.0237960227741265 epoch‚0 % _timestamp‚1.7884406278048227e+09 0loss/FBt1/assignment_loss‚0.7392261713713167 _runtime‚ 344.851922677 /loss/FRt3/assignment_loss‚7.427640175819397 /loss/SRqqt4/assignment_loss‚3.5215370288262 .loss/FRt2/detection_loss‚4.421579944816503 1loss/SRqqt2/detection_loss‚2.2360617520682737u¡‰'“¼‚(:ˆ ³àåÔƒÆÅÇ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÎÔÝ>½‚(:³ ³àåÔëÚ¿É* disk./.usagePercent‚19.422762455237592 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2629.80859375 disk.rbd0.in‚0 disk.dm-1.out‚ 12.140625 disk.drbd1716.out‚0 cpu‚25.045181146083028 proc.cpu.threads‚17 network.sent‚517138 network.recv‚205601 disk.drbd2080.in‚0 disk.sda.out‚ 12.140625 disk.sda3.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 798.44140625* proc.memory.percent‚16.051077842712402$ disk./.usageGB‚289.4131965637207 disk.sda2.in‚0 disk.sda3.out‚ 12.140625 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚977.5947265625# disk.nvme0n1.out‚977.5947265625$ memory_percent‚95.12670040130615 disk.rbd1.out‚ 1.4169921875 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚977.5947265625™…Žž¾‚(:“ ÂàåÔÛ•µÇ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚132.148' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.429599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¾ìB¿‚(:· ÂàåÔÖþ¿É disk.sda2.in‚0' proc.memory.availableMB‚ 798.3359375 cpu‚25.04472017721687) disk./.usagePercent‚19.42442548018002 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 12.62109375 disk.drbd1716.out‚0$ disk.nvme0n1.out‚1022.1923828125 disk.sda2.out‚0 disk.sda3.out‚ 12.62109375 disk.dm-1.out‚ 12.62109375 disk.drbd1716.in‚0! disk.dm-0.out‚1022.1923828125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.recv‚211830 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.54296875 disk.rbd0.out‚0$ memory_percent‚95.12734413146973# proc.memory.rssMB‚ 2629.80859375 proc.cpu.threads‚17$ disk./.usageGB‚289.4379768371582& disk.nvme0c0n1.out‚1022.1923828125 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.051077842712402 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.drbd2080.out‚0 network.sent‚567658 disk.drbd2266.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0b²tÙžÀ‚(:“ ÑàåÔ¡Š¹È gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.017' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.203399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e}2EÁ‚(:º ÑàåÔ•ÓÃÉ' disk.nvme0c0n1.out‚1074.06494140625 disk.sda3.in‚0 disk.sda3.out‚ 13.2421875$ memory_percent‚95.12612819671631# proc.memory.rssMB‚ 2629.80859375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-0.in‚0.078125* proc.memory.percent‚16.051077842712402 network.sent‚573377 disk.rbd1.out‚ 1.54296875 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0( proc.memory.availableMB‚ 798.53515625* disk./.usagePercent‚19.425093659844386 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 network.recv‚215753$ disk./.usageGB‚289.4479331970215 disk.dm-1.out‚ 13.2421875% disk.nvme0n1.out‚1074.06494140625 cpu‚25.044913901285973 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 13.2421875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚1074.06494140625 disk.drbd2266.in‚0Ñ!ÙS0Â+ _wandbruntime‚386  _runtime‚386´”¯Äþ & lr-AdamW/pg1‚5.720762382260494e-05 & lr-AdamW/pg2‚5.720762382260494e-05 trainer/global_step‚449 % _timestamp‚1.7884406697107427e+09 _runtime‚ 386.757968359 _step‚327ªÈÈÄ % _timestamp‚1.7884406697107427e+09 _runtime‚ 386.757968359 _step‚32 & lr-AdamW/pg1‚5.720762382260494e-05 & lr-AdamW/pg2‚5.720762382260494e-05 trainer/global_step‚449 «Ç–0Å+ _wandbruntime‚386  _runtime‚386×0(zSƾO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30905157762560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30502282180354 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 415558933250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30905174932992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚449 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2278244 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚‡>*q"0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30905174932992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213395456 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417937869824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 437805574656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34078573166592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30086740286464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 417937869824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2277695 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7677180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4505772 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9622926 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9622926 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34078597939200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31322313273856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6784019 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9644538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9644538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417155511296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 $ _timestamp‚1.788440669711422e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7677180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 437994197504 _runtime‚ 386.758851425 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31322313273856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31323112802816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4505775 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30086723247104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416340126466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34516592136704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34516378741248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188622848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17321720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7677182 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17300106 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1021 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6783467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31323112802816 =Ù¤È'ÇÂO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7677180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4505772 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9622926 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9622926 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34078597939200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31322313273856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6784019 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9644538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9644538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417155511296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 $ _timestamp‚1.788440669711422e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7677180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 437994197504 _runtime‚ 386.758851425 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31322313273856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31323112802816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4505775 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30086723247104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416340126466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34516592136704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34516378741248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188622848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17321720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7677182 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17300106 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1021 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6783467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31323112802816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30905157762560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30502282180354 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 415558933250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30905174932992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚449 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2278244 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30905174932992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213395456 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417937869824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 437805574656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34078573166592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30086740286464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 417937869824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2277695 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489!ÐäÜ0È+ _wandbruntime‚387  _runtime‚387nLÏ&ÉÉM ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9665898 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17360120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17338480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚ 387.568289174 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2283305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7694220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418865368576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9644260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34154280129536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31391872332800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30973789336064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 438968369152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31392671875072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34593026727936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9644260 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196998144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17338480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9665898 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418082996736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30973806506496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4515767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6798535 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153541036032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30973789336064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31392671875072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1001 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7694222 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34154255356928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7694222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7694220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2282771 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1021 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 438771371008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418865368576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30570822158470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30570023925686 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30973806506496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418082996736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17360120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6799072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6798535 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30153541036032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30973789336064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31392671875072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1001 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7694222 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34154255356928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7694222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7694220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2282771 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1021 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 438771371008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418865368576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30570822158470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30570023925686 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30973806506496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418082996736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17360120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚K[•l.17170432 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6799072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9665898 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17360120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17338480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚ 387.568289174 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2283305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7694220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418865368576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9644260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34154280129536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31391872332800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30973789336064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 438968369152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31392671875072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34593026727936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9644260 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196998144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17338480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9665898 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418082996736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30973806506496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4515767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1"ÐÍog0Ë+ _wandbruntime‚387  _runtime‚387¡ÇhOÌÉ .loss/FBt1/detection_loss‚4.058543878085938 /loss/FRt3/assignment_loss‚7.373926938966263 0loss/SRqqt4/assignment_loss‚3.07723814942116 /loss/FRt1/detection_loss‚2.8263707421054796 1loss/SRqqt1/detection_loss‚2.7310349114589405 /loss/FBt2/detection_loss‚11.352878930313247 %loss/total_loss‚92.51373049078602 epoch‚0 /loss/FRt2/assignment_loss‚7.226219308787379 2loss/SRqqt1/assignment_loss‚3.3124857089098763 1loss/SRqqt3/assignment_loss‚3.483279260429176 1loss/FBt4/assignment_loss‚0.41835803942880234 1loss/SRqqt4/detection_loss‚3.7287768666134324 _step‚35 /loss/FRt4/assignment_loss‚7.307305578915578 2loss/SRqqt2/assignment_loss‚2.1686823377385736 -loss/FRt3/detection_loss‚3.67638612313326 .loss/FBt3/detection_loss‚3.237602583852343 .loss/FBt4/detection_loss‚6.154772672515649 % _timestamp‚1.7884406705222433e+09 0loss/FBt1/assignment_loss‚0.6945695326878474 1loss/FBt3/assignment_loss‚0.24071367956705877 1loss/SRqqt2/detection_loss‚2.5625161011703312 _runtime‚ 387.569353803 -loss/FRt2/detection_loss‚3.36474661991514 .loss/FRt4/detection_loss‚3.103710877164355 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.287726325648172 0loss/FBt2/assignment_loss‚0.5910700909360977 1loss/SRqqt3/detection_loss‚2.5348152330219147/ŒžSÍÍ -loss/FRt2/detection_loss‚3.36474661991514 .loss/FRt4/detection_loss‚3.103710877164355 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.287726325648172 0loss/FBt2/assignment_loss‚0.5910700909360977 1loss/SRqqt3/detection_loss‚2.5348152330219147 .loss/FBt1/detection_loss‚4.058543878085938 /loss/FRt3/assignment_loss‚7.373926938966263 0loss/SRqqt4/assignment_loss‚3.07723814942116 /loss/FRt1/detection_loss‚2.8263707421054796 1loss/SRqqt1/detection_loss‚2.7310349114589405 /loss/FBt2/detection_loss‚11.352878930313247 %loss/total_loss‚92.51373049078602 epoch‚0 /loss/FRt2/assignment_loss‚7.226219308787379 2loss/SRqqt1/assignment_loss‚3.3124857089098763 1loss/SRqqt3/assignment_loss‚3.483279260429176 1loss/FBt4/assignment_loss‚0.41835803942880234 1loss/SRqqt4/detection_loss‚3.7287768666134324 _step‚35 /loss/FRt4/assignment_loss‚7.307305578915578 2loss/SRqqt2/assignment_loss‚2.1686823377385736 -loss/FRt3/detection_loss‚3.67638612313326 .loss/FBt3/detection_loss‚3.237602583852343 .loss/FBt4/detection_loss‚6.154772672515649 % _timestamp‚1.7884406705222433e+09 0loss/FBt1/assignment_loss‚0.6945695326878474 1loss/FBt3/assignment_loss‚0.24071367956705877 1loss/SRqqt2/detection_loss‚2.5625161011703312 _runtime‚ 387.569353803#ƒ¤Ž “΂(:ˆ ààåÔįÈÇ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02F8Ï‚(:­ ààåÔÅ×åÉ* proc.memory.percent‚16.051077842712402 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1115.65478515625% disk.nvme0n1.out‚1115.65478515625 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.87109375 disk.sda2.in‚0 disk.dm-1.out‚13.78125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.sda3.out‚13.78125 disk.dm-0.in‚0.078125$ memory_percent‚95.12557983398438) disk./.usagePercent‚19.42567351690944 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0# proc.memory.availableMB‚798.625 network.sent‚579795 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0" disk.dm-0.out‚1115.65478515625 disk.drbd2266.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2629.80859375 cpu‚25.04566440560387 network.recv‚220761$ disk./.usageGB‚289.4565734863281 disk.sda.out‚13.78125 disk.dm-1.in‚0 disk.drbd2080.out‚0’„“Ђ(:ˆ ïàåÔìˤÇ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚189.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª-ߤDÑ‚(:¹ ïàåÔÉ–Ê disk.dm-0.in‚0.078125 disk.sda.out‚ 14.31640625 disk.drbd1716.out‚0' proc.memory.availableMB‚ 798.1484375* proc.memory.percent‚16.051077842712402 network.sent‚629744 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚1175.0615234375 disk.drbd1716.in‚0 network.recv‚226286$ disk./.usageGB‚289.4772071838379! disk.dm-0.out‚1175.0615234375 disk.drbd2265.out‚0 disk.sda3.out‚ 14.31640625 proc.cpu.threads‚17* disk./.usagePercent‚19.427058261662147 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚1175.0615234375$ memory_percent‚95.12848854064941 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 14.31640625# proc.memory.rssMB‚ 2629.80859375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 cpu‚25.045790111422683 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 1.87109375 disk.sda2.out‚0€÷Pv“Ò‚(:ˆ þàåÔ€ž½Ç gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷BAUFÓ‚(:» þàåÔ”¸É network.sent‚635348 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0' disk.nvme0c0n1.out‚1219.99658203125# proc.memory.rssMB‚ 2629.80859375* proc.memory.percent‚16.051077842712402 network.recv‚230524 disk.dm-1.out‚ 14.74609375 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 14.74609375" disk.dm-0.out‚1219.99658203125 disk.drbd2265.out‚0 disk.rbd0.out‚0$ memory_percent‚95.12708187103271$ disk./.usageGB‚289.4467468261719 disk.dm-0.in‚0.078125 disk.sda1.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 798.37890625 cpu‚25.04544153987433 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 14.74609375 disk.drbd1716.in‚0) disk./.usagePercent‚19.42501404150124% disk.nvme0n1.out‚1219.99658203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.11328125 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125íæéÌ0Ô+ _wandbruntime‚429  _runtime‚429\ÀYÐÃÕ½ & lr-AdamW/pg1‚6.357818326832933e-05 & lr-AdamW/pg2‚6.357818326832933e-05 trainer/global_step‚499 % _timestamp‚1.7884407124919212e+09 _runtime‚ 429.53912618 _step‚36^(õÇÖÁ & lr-AdamW/pg1‚6.357818326832933e-05 & lr-AdamW/pg2‚6.357818326832933e-05 trainer/global_step‚499 % _timestamp‚1.7884407124919212e+09 _runtime‚ 429.53912618 _step‚36$Bûbw0×+ _wandbruntime‚429  _runtime‚429®5‹¹È'ØÂO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 464326555648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7538889 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8529180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34801080163840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34800280634880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34336753608192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19243772 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 463544197120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5005375 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38349149771776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33427612693504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 $ _timestamp‚1.788440712492597e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34336753608192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8529180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2533514 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34336736437760 trainer/global_step‚499 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 463544197120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10714590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10714590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2533003 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8529182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5005372 _step‚37 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10692978 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 464326555648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38349361070080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 486653614080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8529182 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7538375 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33890181270738 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692978 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120Y£Tíc!ÙÆO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34800280634880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34336753608192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19243772 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 463544197120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5005375 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38349149771776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33427612693504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 $ _timestamp‚1.788440712492597e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34336753608192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8529180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2533514 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34336736437760 trainer/global_step‚499 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 463544197120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10714590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10714590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2533003 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8529182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5005372 _step‚37 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10692978 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 464326555648 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38349361070080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 486653614080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8529182 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7538375 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33890181270738 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692978 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 464326555648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7538889 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8529180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34801080163840 IÓÑ&ÛËM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7553970 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1021 trainer/global_step‚499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10735950 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19260532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33957124783494 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19282172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2538603 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 _step‚38 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5015367 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 $ _timestamp‚1.788440713291936e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 465254054400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34405385181696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33494430482432 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8546222 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1001 _runtime‚ 430.33996991 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19282172 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38425783576064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34405385181696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5015364 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7553430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10735950 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264È]r_Õ&ÜÏM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38426013735424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230159360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34405368011264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8546222 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37938389646336 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 487624089088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 462694301062 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7553970 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1021 trainer/global_step‚499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10735950 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19260532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33957124783494 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19282172 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2538603 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 _step‚38 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5015367 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 $ _timestamp‚1.788440713291936e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 465254054400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34405385181696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33494430482432 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8546222 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1001 _runtime‚ 430.33996991 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19282172 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38425783576064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34405385181696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5015364 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7553430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10735950 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Bá‚(:³ áåÔ”ŹÉ* proc.memory.percent‚16.051101684570312& disk.nvme0c0n1.out‚1275.7861328125 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 15.52734375 network.recv‚234333* disk./.usagePercent‚19.426625608929278# disk./.usageGB‚289.470760345459 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda1.out‚0$ disk.nvme0n1.out‚1275.7861328125 disk.dm-0.in‚0.078125 disk.sda.in‚0! disk.dm-0.out‚1275.7861328125 disk.drbd2266.out‚0$ memory_percent‚95.12758255004883 proc.memory.rssMB‚ 2629.8125 cpu‚25.045042541769888 disk.sda.out‚ 15.52734375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.11328125& proc.memory.availableMB‚ 798.296875 network.sent‚640968 disk.sda3.out‚ 15.52734375 disk.drbd2265.in‚0:›¶žâ‚(:“ œáåÔ¬ÂÁÇ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.700599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÊ:ã‚(:¯ œáåÔ±öÒÉ$ memory_percent‚95.12665271759033 proc.cpu.threads‚17 disk.sda.out‚16.03125 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚1326.4736328125 disk.dm-1.out‚16.03125 network.recv‚240566% disk./.usageGB‚289.48863983154297 disk.rbd1.in‚0 disk.rbd1.out‚ 2.3564453125 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd2080.out‚0 network.sent‚690916 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚0.078125! disk.dm-0.out‚1326.4736328125 disk.drbd2266.out‚0( proc.memory.availableMB‚ 798.44921875 proc.memory.rssMB‚ 2629.8125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda2.out‚0 cpu‚25.04517761935225 disk.drbd1716.in‚0 disk.sda3.in‚0 disk.sda3.out‚16.03125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.051101684570312& disk.nvme0c0n1.out‚1326.4736328125* disk./.usagePercent‚19.427825516242265 disk.drbd2080.in‚0쵞ԓä‚(:ˆ «áåÔ¬¹¬Ç gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2758 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ÑAå‚(:¶ «áåÔµ‡¹É disk.drbd2266.in‚0 disk.rbd0.in‚0 network.recv‚245039 disk.sda2.in‚0$ disk.nvme0n1.out‚1365.6318359375! disk.dm-0.out‚1365.6318359375 proc.cpu.threads‚17% disk./.usageGB‚289.56106185913086 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.051125526428223 disk.nvme0c0n1.in‚0.078125& disk.nvme0c0n1.out‚1365.6318359375 cpu‚25.044836744505524 network.sent‚697126 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 16.5703125 disk.dm-1.out‚ 16.5703125 disk.rbd1.out‚ 2.3564453125# proc.memory.rssMB‚ 2629.81640625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ memory_percent‚95.12672424316406$ proc.memory.availableMB‚798.4375* disk./.usagePercent‚19.432685819279815 disk.sda.in‚0 disk.sda.out‚ 16.5703125æm­0æ+ _wandbruntime‚472  _runtime‚472æ@Âç¼ % lr-AdamW/pg1‚6.99487427140537e-05 % lr-AdamW/pg2‚6.99487427140537e-05 trainer/global_step‚549 % _timestamp‚1.7884407550849018e+09 _runtime‚ 472.132096012 _step‚40¤ÌpÆèÀ % _timestamp‚1.7884407550849018e+09 _runtime‚ 472.132096012 _step‚40 % lr-AdamW/pg1‚6.99487427140537e-05 % lr-AdamW/pg2‚6.99487427140537e-05 trainer/global_step‚549(6 ™.0é+ _wandbruntime‚472  _runtime‚472§»Éº êÃO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41646792200192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36768519179264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8292906 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41646816972800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36768502139904 _step‚41 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11763030 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 510715241472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 509932882944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 510715241472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21165824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 535356692480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚549 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11784642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507981756066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9381182 A8DeviceStatsMonitor.on_train_batch_start/nuž‚£Æm_device_alloc‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9381180 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38279047524864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37768332283392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 535159681024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21144210 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37768315112960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38278247995904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1028 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38278247995904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11763030 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 510715241472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21165824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 535356692480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚549 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11784642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507981756066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9381182 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9381180 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38279047524864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37768332283392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 535159681024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21144210 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37768315112960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38278247995904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1028 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38278247995904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11763030 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470)†,u90ì+ _wandbruntime‚472  _runtime‚472Õ“äÒ&íÌM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2792449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1028 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42258830299648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37836946686464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 536144624128 _step‚42 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9398222 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510860368384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38348606597120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36835336968192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 509686905894 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37344225641302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36835319928832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5514964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11784364 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37836963856896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11806002 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42258619014656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186512384 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21204224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9398220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21204224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5514967 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37345023874086 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9398222 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 $ _timestamp‚1.788440755891094e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21182584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508905712470 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8307943 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11784364 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37836963856896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 trainer/global_step‚549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510860368384 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8307413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41722499163136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38347807054848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521Ö 2HîÐM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37836963856896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11806002 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42258619014656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186512384 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21204224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9398220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21204224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5514967 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37345023874086 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9398222 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 $ _timestamp‚1.788440755891094e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21182584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508905712470 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8307943 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11784364 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37836963856896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 trainer/global_step‚549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510860368384 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8307413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41722499163136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38347807054848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2792449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1028 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42258830299648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37836946686464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 536144624128 _step‚42 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9398222 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510860368384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38348606597120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36835336968192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 509686905894 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37344225641302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36835319928832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5514964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11784364*)1þ‚0ï+ _wandbruntime‚472  _runtime‚472îä¥NðÈ 1loss/SRqqt1/detection_loss‚1.1608771633472852 2loss/SRqqt4/assignment_loss‚2.7383246354337007 0loss/SRqqt3/detection_loss‚3.600593252686608 trainer/global_step‚549 0loss/FBt2/assignment_loss‚0.5920793132733225 .loss/FRt4/detection_loss‚2.919906303859674 /loss/FRt2/assignment_loss‚7.143349356121487 .loss/FRt4/assignment_loss‚7.35463977777041 1loss/FBt4/assignment_loss‚0.39518908463651314 epoch‚0 /loss/FRt1/assignment_loss‚7.158496655915913 .loss/FRt2/detection_loss‚4.567884564399719 1loss/SRqqt4/detection_loss‚2.8496472188886606 .loss/FBt3/detection_loss‚2.562430049507384 .loss/FBt4/detection_loss‚3.268200256291924 /loss/FRt3/assignment_loss‚7.363308710210464 1loss/FBt3/assignment_loss‚0.25279643686603365 .loss/FRt3/detection_loss‚4.785507324425613 %loss/total_loss‚92.36493176853867 % _timestamp‚1.7884407558927429e+09 _runtime‚ 472.939865364 0loss/FBt1/assignment_loss‚0.9800873465008206 1loss/SRqqt2/detection_loss‚3.7107265442609787 .loss/FBt1/detection_loss‚8.776207295516693 .loss/FBt2/detection_loss‚9.042782299658832 2loss/SRqqt1/assignment_loss‚2.7151758148245615 1loss/SRqqt2/assignment_loss‚2.304664871909402 1loss/SRqqt3/assignment_loss‚3.370730620164138 .loss/FRt1/detection_loss‚2.751326872068539 _step‚43Ö²åbRñÌ 2loss/SRqqt4/assignment_loss‚2.7383246354337007 0loss/SRqqt3/detection_loss‚3.600593252686608 trainer/global_step‚549 0loss/FBt2/assignment_loss‚0.5920793132733225 .loss/FRt4/detection_loss‚2.919906303859674 /loss/FRt2/assignment_loss‚7.143349356121487 .loss/FRt4/assignment_loss‚7.35463977777041 1loss/FBt4/assignment_loss‚0.39518908463651314 epoch‚0 /loss/FRt1/assignment_loss‚7.158496655915913 .loss/FRt2/detection_loss‚4.567884564399719 1loss/SRqqt4/detection_loss‚2.8496472188886606 .loss/FBt3/detection_loss‚2.562430049507384 .loss/FBt4/detection_loss‚3.268200256291924 /loss/FRt3/assignment_loss‚7.363308710210464 1loss/FBt3/assignment_loss‚0.25279643686603365 .loss/FRt3/detection_loss‚4.785507324425613 %loss/total_loss‚92.36493176853867 % _timestamp‚1.7884407558927429e+09 _runtime‚ 472.939865364 0loss/FBt1/assignment_loss‚0.9800873465008206 1loss/SRqqt2/detection_loss‚3.7107265442609787 .loss/FBt1/detection_loss‚8.776207295516693 .loss/FBt2/detection_loss‚9.042782299658832 2loss/SRqqt1/assignment_loss‚2.7151758148245615 1loss/SRqqt2/assignment_loss‚2.304664871909402 1loss/SRqqt3/assignment_loss‚3.370730620164138 .loss/FRt1/detection_loss‚2.751326872068539 _step‚43 1loss/SRqqt1/detection_loss‚1.1608771633472852+µœð‘ò‚(:† ºáåÔûÒìÇ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ö,4ó‚(:© ºáåÔÏ»ÃÉ network.sent‚703767 disk.drbd2265.in‚0 disk.sda2.out‚0 cpu‚25.04498119535235 network.recv‚250216& disk.nvme0c0n1.out‚1421.1630859375 disk.sda.in‚0( proc.memory.availableMB‚ 798.33984375 proc.cpu.threads‚17 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.12732028961182$ disk./.usageGB‚289.5895538330078 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125# proc.memory.rssMB‚ 2629.81640625 disk.dm-1.out‚17.25* disk./.usagePercent‚19.434597939553058! disk.dm-0.out‚1421.1630859375 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 2.5126953125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.out‚17.25 disk.drbd1716.in‚0 disk.sda.out‚17.25 disk.dm-1.in‚0$ disk.nvme0n1.out‚1421.1630859375 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.051125526428223ÎŽ'ƒ“ô‚(:ˆ ÉáåÔúáÄÇ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì÷3ÅGõ‚(:¼ ÉáåÔ¹”«É cpu‚25.044665081697733 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚753162 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.out‚ 2.5126953125 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 17.76953125& disk.nvme0c0n1.out‚1484.0634765625 disk.sda2.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.051125526428223! disk.dm-0.out‚1484.0634765625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125% disk./.usageGB‚289.61787033081055 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.76953125# proc.memory.rssMB‚ 2629.81640625 network.recv‚255228 disk.sda3.in‚0$ memory_percent‚95.12341022491455 proc.cpu.threads‚17 disk.sda3.out‚ 17.76953125) disk./.usagePercent‚19.43649828347972$ disk.nvme0n1.out‚1484.0634765625 disk.drbd2266.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 798.980468759Î8ö‚(:’ ØáåÔé—ÁÇ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.414' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.68279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iB÷‚(:· ØáåÔì µÊ$ disk./.usageGB‚289.6540641784668 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.1298828125 disk.sda.in‚0 disk.sda3.out‚ 18.3359375 disk.drbd2266.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 18.3359375 network.sent‚759613& disk.nvme0c0n1.out‚1535.6259765625 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚19.438927282964595 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 18.3359375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.04471032117323$ memory_percent‚95.12655735015869 disk.rbd0.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2629.8203125* proc.memory.percent‚16.051149368286133 network.recv‚260019$ disk.nvme0n1.out‚1535.6259765625! disk.dm-0.out‚1535.6259765625( proc.memory.availableMB‚ 798.46484375šÓ^0ø+ _wandbruntime‚514  _runtime‚514†ª,Ãù½ & lr-AdamW/pg2‚7.631930215977809e-05 trainer/global_step‚599 $ _timestamp‚1.788440797793501e+09 _runtime‚ 514.840727273 _step‚44 & lr-AdamW/pg1‚7.631930215977809e-05Æ@"ÇúÁ _runtime‚ 514.840727273 _step‚44 & lr-AdamW/pg1‚7.631930215977809e-05 & lr-AdamW/pg2‚7.631930215977809e-05 trainer/global_step‚599 $ _timestamp‚1.788440797793501e+09,5Îå0û+ _wandbruntime‚514  _runtime‚514]ÒÍ'üÇO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1010 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23087876 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10233180 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 584087758336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 557103927296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41199910958592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41756215356928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41756215356928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12833082 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45430926489600 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41757014885888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45430901716992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 514.841748701 _step‚45 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12833082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6004575 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 554193167474 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41757014885888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10233182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46014798755328 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 554974360690 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10233182 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12854694 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 557103927296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3042663 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9047238 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9046712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556321568768 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40664382986354 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46015014247936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23066262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41199893788160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6004572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521==(Ñ'ýËO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556321568768 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40664382986354 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46015014247936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23066262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41199893788160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6004572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1010 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23087876 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10233180 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 584087758336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 557103927296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41199910958592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41756215356928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41756215356928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12833082 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45430926489600 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41757014885888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45430901716992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 514.841748701 _step‚45 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12833082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6004575 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 554193167474 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41757014885888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10233182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46014798755328 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 554974360690 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10233182 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12854694 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 557103927296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3042663 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9047238 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9046712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0-K^™R0þ+ _wandbruntime‚515  _runtime‚515á-bæîÿÏM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12876054 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ;4DeviceSÆ$0Cç%tatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12876054 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1030 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 555117123878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41268542532096 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3047637 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558031426048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46091666513920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 201192448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6014564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45506583907328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41268525361664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12854416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23104636 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9061674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41826573958144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558031426048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46091440548864 _step‚46 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23104636 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41826573958144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40732124731894 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 557249054208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41268525361664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9062204 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 557249054208 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585057833984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10250220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10250220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40731326499110 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23126276 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40176226414592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45506608679936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592M®¸úÙ&€ÓM $ _timestamp‚1.788440798602522e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10250222 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3047110 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41268542532096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 557249054208 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585057833984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10250220 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10250220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40731326499110 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23126276 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40176226414592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45506608679936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12876054 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12876054 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1030 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 555117123878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41268542532096 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3047637 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558031426048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46091666513920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 201192448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6014564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45506583907328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41268525361664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12854416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23104636 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9061674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41826573958144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558031426048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46091440548864 _step‚46 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23104636 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41826573958144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40732124731894 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 557249054208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41268525361664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9062204 O2„󞆂(:“ öáåÔ≥Ç gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó˜°?B‡‚(:· öáåÔΚ Ê$ memory_percent‚95.12393474578857 network.sent‚816136 disk.rbd0.in‚0 disk.sda.out‚ 19.4609375 disk.dm-1.in‚0 disk.drbd1716.out‚0$ disk.nvme0n1.out‚1643.8818359375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 cpu‚25.04500598249323$ disk./.usageGB‚289.7312927246094 disk.nvme0n1.in‚0.078125 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 19.4609375 disk.drbd2080.in‚0* proc.memory.percent‚16.051149368286133 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚1643.8818359375 disk.drbd1716.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 798.89453125" proc.memory.rssMB‚ 2629.8203125 network.recv‚270577* disk./.usagePercent‚19.444110155495288 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 19.4609375! disk.dm-0.out‚1643.8818359375 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.9892578125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0·Mx“ˆ‚(:ˆ …âåÔ“ŒÈÇ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/¨%™@‰‚(:µ …âåÔŠ¦ÚÊ disk.rbd1.out‚ 3.9892578125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2629.8203125 disk.nvme0c0n1.in‚0.078125 disk.sda1.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.051149368286133 disk.dm-1.out‚ 20.1796875 network.sent‚821858 disk.drbd2265.in‚0 disk.rbd0.out‚0 cpu‚25.04509387014551* disk./.usagePercent‚19.445563766274706! disk.dm-0.out‚1705.7314453125 disk.drbd2080.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 798.890625& disk.nvme0c0n1.out‚1705.7314453125 disk.sda.in‚0 disk.sda.out‚ 20.1796875 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚289.7529525756836 disk.sda3.out‚ 20.1796875 disk.drbd1716.in‚0$ disk.nvme0n1.out‚1705.7314453125 disk.drbd2266.in‚0 disk.rbd1.in‚0 network.recv‚274681 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚95.12395858764648ͧÂß0Š+ _wandbruntime‚557  _runtime‚557j8œÄ‹¾ trainer/global_step‚649 % _timestamp‚1.7884408401697953e+09 _runtime‚ 557.216992425 _step‚48 & lr-AdamW/pg1‚8.268986160550246e-05 & lr-AdamW/pg2‚8.268986160550246e-05Zã,êȌ & lr-AdamW/pg1‚8.268986160550246e-05 & lr-AdamW/pg2‚8.268986160550246e-05 trainer/global_step‚649 % _timestamp‚1.7884408401697953e+09 _runtime‚ 557.216992425 _step‚480·‘Ü0+ _wandbruntime‚557  _runtime‚557Qºg ‘ŽÇO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 600404578882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13924746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45234982246912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11085182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9801696 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13924746 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43450281032704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24988314 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11085182 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49847806874112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6504175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49847587187200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44631489633792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11085180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49215011233792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 602710254592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884408401704483e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 603492613120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45234182717952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 632575953408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6504172 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 trainer/global_step‚649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3298059 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3297524 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 632770867712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194914304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43450298072064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.curreÍÑL9<nt‚ 781193216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9802234 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45234182717952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13903134 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11085180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13903134 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1030 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010ûfÜËÑ'ËO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 600404578882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13924746 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45234982246912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11085182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9801696 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13924746 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43450281032704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24988314 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11085182 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49847806874112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6504175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49847587187200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44631489633792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11085180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49215011233792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 602710254592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884408401704483e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 603492613120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45234182717952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 632575953408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6504172 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 trainer/global_step‚649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3298059 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3297524 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 632770867712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194914304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43450298072064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9802234 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45234182717952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13903134 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11085180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13903134 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1030 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚10101çÄ0+ _wandbruntime‚558  _runtime‚558x¯L×&‘ÑM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44700104036864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44700104036864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25048328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 604420111872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1030 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49924470293504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633573972992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 604420111872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13924468 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49290693424128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9817314 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45304541319168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49924267397120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202896384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43517098821632 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 602109728710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6514164 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _runtime‚ 558.012125763 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45303741776896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9816780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25026688 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 603637740032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11102220 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6514167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11102220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178123776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44700121207296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 603637740032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13946106 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3302616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328€9ýÉ’ÕM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45304541319168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49924267397120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202896384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43517098821632 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 602109728710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6514164 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _runtime‚ 558.012125763 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45303741776896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9816780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25026688 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 603637740032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11102220 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6514167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11102220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178123776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44700121207296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 603637740032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13946106 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3302616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44700104036864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44700104036864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25048328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 604420111872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1030 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49924470293504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633573972992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 604420111872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13924468 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49290693424128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9817314 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚5212HÙ”¹0“+ _wandbruntime‚558  _runtime‚558“p‚‰N”È .loss/FRt3/detection_loss‚4.066483841522744 .loss/FRt4/detection_loss‚3.433316331842671 /loss/SRqqt4/detection_loss‚2.34060325473547 /loss/FBt3/detection_loss‚1.6745067534332365 % _timestamp‚1.7884408409659321e+09 _step‚51 0loss/FRt2/assignment_loss‚7.2434243361155195 /loss/FRt4/assignment_loss‚7.397425599720167 1loss/SRqqt4/assignment_loss‚2.473123882897198 0loss/FBt3/assignment_loss‚0.1626627131520094 /loss/FRt1/detection_loss‚2.7402263984636024 1loss/SRqqt1/detection_loss‚1.4537426880587143 /loss/FRt3/assignment_loss‚7.387676791140907 -loss/FBt2/detection_loss‚37.3707313016057 .loss/FBt4/detection_loss‚8.212436376041488 .loss/FRt2/detection_loss‚6.700097292661667 0loss/FBt1/assignment_loss‚0.7550745290962368 0loss/FBt2/assignment_loss‚0.6095207987164031 epoch‚0 /loss/FRt1/assignment_loss‚7.184420832881221 2loss/SRqqt2/assignment_loss‚2.7370834059493485 1loss/SRqqt3/assignment_loss‚3.387381734270038 1loss/SRqqt2/detection_loss‚3.7589024977628576 1loss/SRqqt3/detection_loss‚2.0411559553885157 trainer/global_step‚649 2loss/SRqqt1/assignment_loss‚2.5978373791042126 %loss/total_loss‚121.9912101085408 _runtime‚ 558.013038556 0loss/FBt4/assignment_loss‚0.3934739126891241 .loss/FBt1/detection_loss‚5.869901501291741½g„JR•Ì .loss/FRt2/detection_loss‚6.700097292661667 0loss/FBt1/assignment_loss‚0.7550745290962368 0loss/FBt2/assignment_loss‚0.6095207987164031 epoch‚0 /loss/FRt1/assignment_loss‚7.184420832881221 2loss/SRqqt2/assignment_loss‚2.7370834059493485 1loss/SRqqt3/assignment_loss‚3.387381734270038 1loss/SRqqt2/detection_loss‚3.7589024977628576 1loss/SRqqt3/detection_loss‚2.0411559553885157 trainer/global_step‚649 2loss/SRqqt1/assignment_loss‚2.5978373791042126 %loss/total_loss‚121.9912101085408 _runtime‚ 558.013038556 0loss/FBt4/assignment_loss‚0.3934739126891241 .loss/FBt1/detection_loss‚5.869901501291741 .loss/FRt3/detection_loss‚4.066483841522744 .loss/FRt4/detection_loss‚3.433316331842671 /loss/SRqqt4/detection_loss‚2.34060325473547 /loss/FBt3/detection_loss‚1.6745067534332365 % _timestamp‚1.7884408409659321e+09 _step‚51 0loss/FRt2/assignment_loss‚7.2434243361155195 /loss/FRt4/assignment_loss‚7.397425599720167 1loss/SRqqt4/assignment_loss‚2.473123882897198 0loss/FBt3/assignment_loss‚0.1626627131520094 /loss/FRt1/detection_loss‚2.7402263984636024 1loss/SRqqt1/detection_loss‚1.4537426880587143 /loss/FRt3/assignment_loss‚7.387676791140907 -loss/FBt2/detection_loss‚37.3707313016057 .loss/FBt4/detection_loss‚8.2124363760414883BÓî“–‚(:ˆ ”âåÔ¸¹üÇ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ëW6—‚(:« ”âåÔ±Ã†Ê disk.dm-0.out‚1750.267578125 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2629.8203125 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0c0n1.out‚1750.267578125 disk.sda3.out‚20.5625 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚0.078125 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0( proc.memory.availableMB‚ 798.89453125 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.sda2.in‚0 network.recv‚279535 disk.drbd2265.out‚0* proc.memory.percent‚16.051149368286133$ disk./.usageGB‚289.7680892944336 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0$ memory_percent‚95.12393474578857 cpu‚25.04485422458626 disk.dm-1.out‚20.5625 disk.drbd1716.out‚0# disk.nvme0n1.out‚1750.267578125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 4.0517578125 disk.sda3.in‚0 disk.sda.out‚20.5625 network.sent‚828277* disk./.usagePercent‚19.446579604170573 disk.dm-1.in‚0ÑËš`ž˜‚(:“ £âåÔ¹˜™Ç gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚192.978' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.595600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0® D™‚(:¹ £âåÔÿ²£É disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 21.1328125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.0517578125 disk.sda1.out‚0" proc.memory.rssMB‚ 2629.8203125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 cpu‚25.04504703140298 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 network.recv‚286043( disk./.usagePercent‚19.4468220433055 disk.drbd1716.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 798.90234375 disk.rbd0.in‚0 disk.nvme0n1.in‚0.078125$ memory_percent‚95.12388706207275* proc.memory.percent‚16.051149368286133' disk.nvme0c0n1.out‚1806.45068359375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 21.1328125% disk.nvme0n1.out‚1806.45068359375" disk.dm-0.out‚1806.45068359375 proc.cpu.threads‚17 disk.drbd2080.in‚0 network.sent‚878837% disk./.usageGB‚289.77170181274414 disk.sda3.out‚ 21.1328125®G¡ò0š+ _wandbruntime‚599  _runtime‚599I“£Ä›¾ _runtime‚ 599.844984563 _step‚52 & lr-AdamW/pg1‚8.906042105122685e-05 & lr-AdamW/pg2‚8.906042105122685e-05 trainer/global_step‚699 % _timestamp‚1.7884408827977915e+09l’—Ȝ & lr-AdamW/pg1‚8.906042105122685e-05 & lr-AdamW/pg2‚8.906042105122685e-05 trainer/global_step‚699 % _timestamp‚1.7884408827977915e+09 _runtime‚ 599.844984563 _step‚524Ôq¿70+ _wandbruntime‚599  _runtime‚599ót™Ï'žÉO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1012 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46791187518464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48712150078976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 649098940416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14994798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47438584701970 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53680679107072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48063068308992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11937180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53680900891136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46791170479104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14994798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 649881298944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11937182 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 647397183506 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48712150078976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553461 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26910366 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 681558356480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚699 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48063051138560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 649881298944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26931980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10556703 _runtime‚ 599.845876106ߊŸÓ'ŸÍO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48712150078976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553461 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26910366 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 681558356480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚699 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48063051138560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 649881298944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26931980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10556703 _runtime‚ 599.845876106 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1012 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46791187518464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48712150078976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 649098940416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14994798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47438584701970 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53680679107072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48063068308992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11937180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53680900891136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46791170479104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14994798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 649881298944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11937182 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 647397183506 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10571821 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytйM|es.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48131699882496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48782508680192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26970380 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11954222 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46857988268032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 682535392256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47505528214726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 650026425856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 648321140118 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788440883607244e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48131699882496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11954222 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14994520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26970380 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512D·«BÜ&¤ÖM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48131699882496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48782508680192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26970380 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11954222 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46857988268032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 682535392256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47505528214726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 650026425856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 648321140118 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788440883607244e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48131699882496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11954222 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14994520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26970380 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15016158 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11954220 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15016158 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 682724001792 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 650808797696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 647539946694 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7013767 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 650026425856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10571821 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚06?’B^0¥+ _wandbruntime‚600  _runtime‚600ÃB\M¦Ç /loss/FRt3/assignment_loss‚7.399597563062396 1loss/SRqqt4/assignment_loss‚2.855765476822853 1loss/FBt2/assignment_loss‚0.37733219761101183 .loss/FRt3/detection_loss‚4.386379362855639 0loss/SRqqt3/detection_loss‚4.204763622752149 epoch‚0 2loss/SRqqt2/assignment_loss‚2.4825357489096813 % _timestamp‚1.7884408836089196e+09 /loss/FRt4/assignment_loss‚7.331609597281804 0loss/FBt1/assignment_loss‚0.6875495770398308 0loss/FBt4/assignment_loss‚0.2753148201403459 .loss/FRt4/detection_loss‚2.494917325912014 1loss/SRqqt1/detection_loss‚1.5341159773570037 0loss/SRqqt2/detection_loss‚4.043305871960444 _runtime‚ 600.656071555 /loss/FRt1/assignment_loss‚7.227955750056675 0loss/FBt3/assignment_loss‚0.2581805940238151 0loss/SRqqt4/detection_loss‚2.388623026199639 .loss/FBt4/detection_loss‚3.844047795392965 1loss/SRqqt3/assignment_loss‚3.244112160470751 .loss/FRt1/detection_loss‚2.205631487248909 .loss/FBt2/detection_loss‚7.738670510681052 /loss/FBt3/detection_loss‚2.5829940224905457 _step‚55 0loss/FRt2/assignment_loss‚7.0686138788859045 %loss/total_loss‚87.03608147960392 1loss/SRqqt1/assignment_loss‚2.741606312493483 .loss/FRt2/detection_loss‚3.538897665341695 .loss/FBt1/detection_loss‚6.123561134613309 trainer/global_step‚699CÙìÿQ§Ë 1loss/SRqqt3/assignment_loss‚3.244112160470751 .loss/FRt1/detection_loss‚2.205631487248909 .loss/FBt2/detection_loss‚7.738670510681052 /loss/FBt3/detection_loss‚2.5829940224905457 _step‚55 0loss/FRt2/assignment_loss‚7.0686138788859045 %loss/total_loss‚87.03608147960392 1loss/SRqqt1/assignment_loss‚2.741606312493483 .loss/FRt2/detection_loss‚3.538897665341695 .loss/FBt1/detection_loss‚6.123561134613309 trainer/global_step‚699 /loss/FRt3/assignment_loss‚7.399597563062396 1loss/SRqqt4/assignment_loss‚2.855765476822853 1loss/FBt2/assignment_loss‚0.37733219761101183 .loss/FRt3/detection_loss‚4.386379362855639 0loss/SRqqt3/detection_loss‚4.204763622752149 epoch‚0 2loss/SRqqt2/assignment_loss‚2.4825357489096813 % _timestamp‚1.7884408836089196e+09 /loss/FRt4/assignment_loss‚7.331609597281804 0loss/FBt1/assignment_loss‚0.6875495770398308 0loss/FBt4/assignment_loss‚0.2753148201403459 .loss/FRt4/detection_loss‚2.494917325912014 1loss/SRqqt1/detection_loss‚1.5341159773570037 0loss/SRqqt2/detection_loss‚4.043305871960444 _runtime‚ 600.656071555 /loss/FRt1/assignment_loss‚7.227955750056675 0loss/FBt3/assignment_loss‚0.2581805940238151 0loss/SRqqt4/detection_loss‚2.388623026199639 .loss/FBt4/detection_loss‚3.8440477953929657¸’Qt“¨‚(:ˆ ÁâåÔíã©Ç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚194.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.9874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„>é†B©‚(:· ÁâåÔ¦ŒÊ disk.sda3.out‚ 22.140625 disk.drbd1716.out‚0" disk.dm-0.out‚1917.41650390625" proc.memory.rssMB‚ 2629.8203125 disk.drbd2080.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0* disk./.usagePercent‚19.446547859236325 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚1917.41650390625' proc.memory.availableMB‚ 798.8515625 network.recv‚295805% disk./.usageGB‚289.76761627197266 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 4.193359375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 22.140625 disk.dm-0.in‚0.078125$ memory_percent‚95.12419700622559 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.sent‚923409 disk.rbd1.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.051149368286133 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 22.140625 disk.sda2.in‚0 disk.drbd2266.out‚0 cpu‚25.045212406064806 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1917.41650390625¢m ´žª‚(:“ ÐâåÔ×Ù–Ç gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.885000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Ë$?«‚(:´ ÐâåÔ·—Ê( proc.memory.availableMB‚ 798.60546875 disk.sda3.out‚22.59375 disk.sda.out‚22.59375 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚289.7902069091797 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 4.255859375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2629.8203125 disk.sda.in‚0* disk./.usagePercent‚19.448063935854094% disk.nvme0n1.out‚1970.77880859375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1970.77880859375 disk.dm-1.in‚0 network.sent‚962539 network.recv‚301352 disk.nvme0c0n1.in‚0.078125 disk.dm-1.out‚22.59375$ memory_percent‚95.12569904327393* proc.memory.percent‚16.051149368286133 cpu‚25.045315626618358" disk.dm-0.out‚1970.77880859375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.out‚0p‡*z0¬+ _wandbruntime‚642  _runtime‚642ŸÏ»µÄ­¾ & lr-AdamW/pg2‚9.543098049695122e-05 trainer/global_step‚749 % _timestamp‚1.7884409252387977e+09 _runtime‚ 642.286284171 _step‚56 & lr-AdamW/pg1‚9.543098049695122e-05øÚ¢NȮ % _timestamp‚1.7884409252387977e+09 _runtime‚ 642.286284171 _step‚56 & lr-AdamW/pg1‚9.543098049695122e-05 & lr-AdamW/pg2‚9.543098049695122e-05 trainer/global_step‚7498ßš¡Á0¯+ _wandbruntime‚642  _runtime‚642‡#WÅÚ!°ÈO ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 696269984768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7503372 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12789180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52190117440000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 $ _timestamp‚1.788440925239672e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28854032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11313667 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3809756 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 696269984768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51494629813760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12789182 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56783255040000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 693608594914 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51494629813760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11313128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52190117440000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57513836464128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50132059925504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57513625165824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 730581424128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52190916968960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16064850 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52190117440000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 $ _timestamp‚1.788440925239672e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28854032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11313667 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3809756 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 696269984768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51494629813760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12789182 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56783255040000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 693608594914 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51494629813760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11313128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52190117440000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57513836464128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50132059925504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57513625165824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 730581424128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52190916968960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16064850 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28892432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50198877714432 trainer/global_step‚749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11328259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28892432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57590273191424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7513367 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50198894753792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12806220 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52260476041216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697197483520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11328804 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56858912457728 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50892629072534 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16086210 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57590494962176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51563261387264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28870792 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697197483520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1015 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 694532551526 _step‚58 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16064572 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12806222 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52260476041216 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884409260295591e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52259676498944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7513364 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12806220 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16086210 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÐÛ®$Ý&´×M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7513364 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12806220 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16086210 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28892432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50198877714432 trainer/global_step‚749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221770752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11328259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28892432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57590273191424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7513367 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50198894753792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12806220 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52260476041216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697197483520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11328804 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56858912457728 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50892629072534 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16086210 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57590494962176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51563261387264 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28870792 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697197483520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1015 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 694532551526 _step‚58 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16064572 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12806222 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52260476041216 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884409260295591e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52259676498944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0:d¹o0µ+ _wandbruntime‚643  _runtime‚643ISÔÀó¶Í /loss/FRt4/assignment_loss‚7.276588422280771 1loss/FBt3/assignment_loss‚0.38759580547728917 /loss/FRt1/detection_loss‚2.2017544396221638 /loss/FRt4/detection_loss‚2.7747948553275177 trainer/global_step‚749 /loss/FBt3/detection_loss‚0.9381543972364605 0loss/FRt2/assignment_loss‚7.2335287224162705 /loss/FRt3/assignment_loss‚7.461644445146833 0loss/FBt1/assignment_loss‚0.6780285489720492 .loss/FRt3/detection_loss‚3.553837634268261 1loss/SRqqt2/detection_loss‚3.4406271750150723 0loss/SRqqt3/detection_loss‚4.767715357573123 /loss/FBt2/detection_loss‚6.5787456618113955 2loss/SRqqt2/assignment_loss‚2.8438628801079684 0loss/FBt2/assignment_loss‚0.6552538909053179 % _timestamp‚1.7884409260310323e+09 1loss/SRqqt1/assignment_loss‚2.732942568950164 0loss/SRqqt4/detection_loss‚2.761141725868549 epoch‚0 1loss/FBt4/assignment_loss‚0.30833360269635673 .loss/FRt2/detection_loss‚3.021592552011663 %loss/total_loss‚83.346MÒJ`16163538271 -loss/FBt1/detection_loss‚5.06074028923398 _step‚59 /loss/FRt1/assignment_loss‚7.245075641139861 2loss/SRqqt4/assignment_loss‚2.6399619230684244 1loss/SRqqt1/detection_loss‚1.3328399710550367 2loss/SRqqt3/assignment_loss‚3.2251058908609243 .loss/FBt4/detection_loss‚4.226295234337275 _runtime‚ 643.078161371ÏT÷@W·Ñ /loss/FRt4/assignment_loss‚7.276588422280771 1loss/FBt3/assignment_loss‚0.38759580547728917 /loss/FRt1/detection_loss‚2.2017544396221638 /loss/FRt4/detection_loss‚2.7747948553275177 trainer/global_step‚749 /loss/FBt3/detection_loss‚0.9381543972364605 0loss/FRt2/assignment_loss‚7.2335287224162705 /loss/FRt3/assignment_loss‚7.461644445146833 0loss/FBt1/assignment_loss‚0.6780285489720492 .loss/FRt3/detection_loss‚3.553837634268261 1loss/SRqqt2/detection_loss‚3.4406271750150723 0loss/SRqqt3/detection_loss‚4.767715357573123 /loss/FBt2/detection_loss‚6.5787456618113955 2loss/SRqqt2/assignment_loss‚2.8438628801079684 0loss/FBt2/assignment_loss‚0.6552538909053179 % _timestamp‚1.7884409260310323e+09 1loss/SRqqt1/assignment_loss‚2.732942568950164 0loss/SRqqt4/detection_loss‚2.761141725868549 epoch‚0 1loss/FBt4/assignment_loss‚0.30833360269635673 .loss/FRt2/detection_loss‚3.021592552011663 %loss/total_loss‚83.34616163538271 -loss/FBt1/detection_loss‚5.06074028923398 _step‚59 /loss/FRt1/assignment_loss‚7.245075641139861 2loss/SRqqt4/assignment_loss‚2.6399619230684244 1loss/SRqqt1/detection_loss‚1.3328399710550367 2loss/SRqqt3/assignment_loss‚3.2251058908609243 .loss/FBt4/detection_loss‚4.226295234337275 _runtime‚ 643.078161371;ûæ4“¸‚(:ˆ ßâåÔƒíØÇ gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ÕUC¹‚(:¸ ßâåÔ¨ÉÉ disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.255859375 network.sent‚968545 disk.nvme0c0n1.in‚0.078125" disk.dm-0.out‚2032.18310546875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2032.18310546875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 23.2109375 disk.dm-1.out‚ 23.2109375" proc.memory.rssMB‚ 2629.8203125 disk.sda1.out‚0 disk.sda.out‚ 23.2109375 disk.drbd1716.out‚0 disk.drbd2080.in‚0$ memory_percent‚95.12479305267334 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 798.75390625* proc.memory.percent‚16.051149368286133 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 cpu‚25.04546667144585 disk.dm-0.in‚0.078125 network.recv‚305569 disk.sda1.in‚0* disk./.usagePercent‚19.449137887460235# disk./.usageGB‚289.806209564209 disk.rbd0.in‚0% disk.nvme0n1.out‚2032.18310546875îGŠ8’º‚(:‡ îâåÔì§ÙÇ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.975' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rh05M»‚(: îâåÔèñðÉ disk.drbd2265.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2090.53857421875 network.sent‚ 1.018875e+06$ disk./.usageGB‚289.8272171020508 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚2090.53857421875" disk.dm-0.out‚2090.53857421875 disk.rbd1.in‚0 network.recv‚311667 disk.sda3.out‚ 23.66015625 disk.dm-1.out‚ 23.66015625 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚95.12698650360107( proc.memory.availableMB‚ 798.39453125" proc.memory.rssMB‚ 2629.8203125 cpu‚25.045542534730917 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.3974609375 disk.sda.out‚ 23.66015625* proc.memory.percent‚16.051149368286133 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 proc.cpu.threads‚17( disk./.usagePercent‚19.4505477209513 disk.sda2.in‚0y!„Ï“¼‚(:ˆ ýâåÔ͵®Ç gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.019' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00K½‚(:À ýâåÔç™Ê disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.3974609375 disk.rbd0.in‚0$ memory_percent‚95.12696266174316 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 24.4921875 network.sent‚ 1.024425e+06 disk.drbd2080.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2127.35302734375 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2629.8203125 cpu‚25.045968559803928 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ disk./.usageGB‚289.8564262390137 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125* proc.memory.percent‚16.051149368286133 network.recv‚315818* disk./.usagePercent‚19.452507970640976 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 798.3984375% disk.nvme0n1.out‚2127.35302734375 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda.out‚ 24.4921875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 24.4921875 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚2127.35302734375 disk.sda1.out‚0 proc.cpu.threads‚17näÁ‚0¾+ _wandbruntime‚684  _runtime‚684Ž±ï Æ¿À % _timestamp‚1.7884409678780658e+09 _runtime‚ 684.925269512 _step‚60 ' lr-AdamW/pg1‚0.00010180153994267561 ' lr-AdamW/pg2‚0.00010180153994267561 trainer/global_step‚799fdMuÊÀÄ ' lr-AdamW/pg1‚0.00010180153994267561 ' lr-AdamW/pg2‚0.00010180153994267561 trainer/global_step‚799 % _timestamp‚1.7884409678780658e+09 _runtime‚ 684.925269512 _step‚60<ÕÚ ]0Á+ _wandbruntime‚684  _runtime‚684¿å™HÎ'ÂÈO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 739038813106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17113290 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54926208488960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13641180 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60567339784192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55668084801024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 741876312064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 739820006322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12068778 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17134902 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8002972 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54211988185010 $ _timestamp‚1.788440967878764e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13641182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 171845632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30754470 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13641182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4065803 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60567364556800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54926225659392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17113290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 _step‚61 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54926225659392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55668884329984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61346781232640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55668884329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8002975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55668084801024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779244830208óÛÒÒ'ÃÌO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13641180 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60567339784192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55668084801024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 741876312064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 739820006322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12068778 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17134902 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8002972 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54211988185010 $ _timestamp‚1.788440967878764e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13641182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 171845632 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30754470 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13641182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4065803 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60567364556800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54926225659392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17113290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 _step‚61 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54926225659392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55668884329984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61346781232640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55668884329984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8002975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55668084801024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779244830208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 739038813106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17113290 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54926208488960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0=«Jê0Ä+ _wandbruntime‚685  _runtime‚685† tr†ÅÓM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17134624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61423259915264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60643046747136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17156262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54994857232896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12083332 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8012967 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 743586169344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61423460714496 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54279729930342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13658220 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54994857232896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30814484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55737643859968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54994840062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13658220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 740743962934 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 _runtime‚ 685.722252917 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17134624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30792844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13658222 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53539767160832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17156262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200799232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30814484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176026624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 % _timestamp‚1.7884409686746757e+09Qˆ?pÝ&Æ×M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 743586169344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61423460714496 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54279729930342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13658220 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54994857232896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30814484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55737643859968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54994840062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13658220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 740743962934 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 _runtime‚ 685.722252917 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17134624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30792844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13658222 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53539767160832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17156262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200799232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30814484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176026624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 % _timestamp‚1.7884409686746757e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17134624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61423259915264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60643046747136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17156262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54994857232896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12083332 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8012967 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489>WQ0Ç+ _wandbruntime‚685  _runtime‚685<×#HÈ 1loss/SRqqt3/assignment_loss‚3.144116004308065 1loss/SRqqt4/assignment_loss‚2.561470521820916 1loss/SRqqt2/assignment_loss‚2.744191585552125 0loss/FBt4/assignment_loss‚0.4617586432460008 1loss/SRqqt1/detection_loss‚1.3802910964249937 0loss/SRqqt4/detection_loss‚2.079876652016999 epoch‚0 % _timestamp‚1.7884409686764865e+09 /loss/FRt2/assignment_loss‚7.386218574311998 0loss/FBt2/assignment_loss‚0.3339527284399992 .loss/FBt3/detection_loss‚2.345791077237074 /loss/FBt4/detection_loss‚2.3659140535671233 /loss/FRt1/assignment_loss‚7.197900833622102 /loss/FRt3/assignment_loss‚7.359074115753174 0loss/FBt3/assignment_loss‚0.5346790049745778 0loss/SRqqt2/detection_loss‚3.526341506413051 trainer/global_step‚799 _runtime‚ 685.723634524 0loss/FBt1/assignment_loss‚0.5775781375632505 .loss/FRt4/detection_loss‚2.615410468391409 1loss/SRqqt3/detection_loss‚1.8919066528542974 -loss/FBt1/detection_loss‚6.44890950194427 .loss/FBt2/detection_loss‚4.551390065749486 _step‚63 .loss/FRt2/detection_loss‚4.596600072251426 -loss/FRt3/detection_loss‚4.45702826557681 -loss/FRt4/assignment_loss‚7.3614536183221 2loss/SRqqt1/assignment_loss‚2.7117919276742373 .loss/FRt1/detection_loss‚2.251922591619434 %loss/total_loss‚80.88556769963492<<`hLÉÆ 1loss/SRqqt2/assignment_loss‚2.744191585552125 0loss/FBt4/assignment_loss‚0.4617586432460008 1loss/SRqqt1/detection_loss‚1.3802910964249937 0loss/SRqqt4/detection_loss‚2.079876652016999 epoch‚0 % _timestamp‚1.7884409686764865e+09 /loss/FRt2/assignment_loss‚7.386218574311998 0loss/FBt2/assignment_loss‚0.3339527284399992 .loss/FBt3/detection_loss‚2.345791077237074 /loss/FBt4/detection_loss‚2.3659140535671233 /loss/FRt1/assignment_loss‚7.197900833622102 /loss/FRt3/assignment_loss‚7.359074115753174 0loss/FBt3/assignment_loss‚0.5346790049745778 0loss/SRqqt2/detection_loss‚3.526341506413051 trainer/global_step‚799 _runtime‚ 685.723634524 0loss/FBt1/assignment_loss‚0.5775781375632505 .loss/FRt4/detection_loss‚2.615410468391409 1loss/SRqqt3/detection_loss‚1.8919066528542974 -loss/FBt1/detection_loss‚6.44890950194427 .loss/FBt2/detection_loss‚4.551390065749486 _step‚63 .loss/FRt2/detection_loss‚4.596600072251426 -loss/FRt3/detection_loss‚4.45702826557681 -loss/FRt4/assignment_loss‚7.3614536183221 2loss/SRqqt1/assignment_loss‚2.7117919276742373 .loss/FRt1/detection_loss‚2.251922591619434 %loss/total_loss‚80.88556769963492 1loss/SRqqt3/assignment_loss‚3.144116004308065 1loss/SRqqt4/assignment_loss‚2.561470521820916?+ùžÊ‚(:“ ŒãåÔ·ÔÉÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ=üOË‚(:Ä ŒãåÔãà¿Ê" disk.dm-0.out‚2178.34130859375 disk.sda3.out‚ 24.93359375" proc.memory.rssMB‚ 2629.8203125 network.sent‚ 1.030442e+06 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2266.out‚0 cpu‚25.04568499219055* disk./.usagePercent‚19.453648484206017 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 24.93359375% disk.nvme0n1.out‚2178.34130859375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.4599609375' disk.nvme0c0n1.out‚2178.34130859375 disk.sda.in‚0$ memory_percent‚95.12741565704346% disk./.usageGB‚289.87342071533203 disk.drbd1716.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 798.32421875 network.recv‚320467 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.in‚0 disk.sda.out‚ 24.93359375 disk.sda2.in‚0* proc.memory.percent‚16.051149368286133 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚0÷(+ðžÌ‚(:“ ›ãåÔžØ¤Ç gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 cì*KÍ‚(:À ›ãåÔ÷´©Ê disk.sda.out‚ 25.48828125$ memory_percent‚95.12758255004883% disk.nvme0n1.out‚2245.42626953125$ disk./.usageGB‚289.8939971923828 disk.dm-1.out‚ 25.48828125 disk.drbd2266.out‚0& proc.memory.availableMB‚ 798.296875 network.sent‚ 1.079951e+06 disk.sda2.in‚0" disk.dm-0.out‚2245.42626953125 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 25.48828125 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2629.828125 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚2245.42626953125 disk.rbd1.in‚0 cpu‚25.04578596776945 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.051197052001953 proc.cpu.threads‚17 network.recv‚325514* disk./.usagePercent‚19.455029388845706 disk.sda2.out‚0 disk.rbd1.out‚ 4.7255859375&÷v€žÎ‚(:“ ªãåÔ¼˜¼Ç gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.895799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º§ÂKÏ‚(:À ªãåÔøŸ™Ê disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125! proc.memory.rssMB‚ 2629.828125 cpu‚25.045879130456587% disk./.usageGB‚289.91839599609375 disk.drbd2080.in‚0 disk.dm-1.out‚ 25.88671875* proc.memory.percent‚16.051197052001953 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 25.88671875 disk.drbd2266.out‚0! disk.dm-0.out‚2300.0400390625( proc.memory.availableMB‚ 797.79296875 network.sent‚ 1.086221e+06 network.recv‚330406$ disk.nvme0n1.out‚2300.0400390625 disk.drbd2266.in‚0) disk./.usagePercent‚19.45666681303471 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚95.13065814971924 disk.drbd2265.in‚0 disk.rbd1.out‚ 4.9833984375 disk.nvme0c0n1.in‚0.078125& disk.nvme0c0n1.out‚2300.0400390625 disk.sda.out‚ 25.88671875 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0«ì0Ð+ _wandbruntime‚727  _runtime‚727 çC›ÅÑ¿ trainer/global_step‚849 $ _timestamp‚1.788441010461096e+09 _runtime‚ 727.508294981 _step‚64 ' lr-AdamW/pg1‚0.00010817209938839998 ' lr-AdamW/pg2‚0.00010817209938839998«ÏkÉÒà ' lr-AdamW/pg2‚0.00010817209938839998 trainer/global_step‚849 $ _timestamp‚1.788441010461096e+09 _runtime‚ 727.508294981 _step‚64 ' lr-AdamW/pg1‚0.00010817209938839998@g§0Ó+ _wandbruntime‚727  _runtime‚727üaÃÏ'ÔÉO trainer/global_step‚849 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚ 727.509180221 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14493182 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 828452013056 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57599089042818 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58357804334592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 789047356416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884410104617712e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4322322 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8502575 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58357787164160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32676522 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 788264997888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32698136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1015 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56813838818304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58357804334592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14493182 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18204954 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58357787164160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 788264997888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18183342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12824375 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14493180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59146052162048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64351449300992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59146851691008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57599887275394 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59146052162048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 828269681664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 789047356416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58357804334592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 TDDeviceStatsMonitor.›bܳº&on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14493182 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18204954 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58357787164160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 788264997888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18183342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12824375 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14493180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59146052162048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64351449300992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59146851691008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57599887275394 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59146052162048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 828269681664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 789047356416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1015 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56813838818304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389A븫c0Ö+ _wandbruntime‚728  _runtime‚728~°AÙ&×ÓM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14510222 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1015 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56880656607232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8512564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58426418737664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65256596655616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 829440391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57666830788150 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8512567 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 786174180918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32736536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4326919 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14510220 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57667629020934 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32714896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12840015 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64427156263936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64427131491328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32736536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59216410763264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32714896 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚66 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18226314 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56880673646592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 789192483328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14510222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4327448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 % _timestamp‚1.7884410112612815e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58426435908096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 789974855168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18226314 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 786955374342 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12839483 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184415232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65256387467776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14510220 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18204676 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 829255976448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58426435908096 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 789192483328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59215611220992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209187840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010Ì¿Ý&Ø×M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59215611220992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209187840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14510222 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1015 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56880656607232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8512564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58426418737664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65256596655616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 829440391680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57666830788150 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1035 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8512567 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 786174180918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32736536 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4326919 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14510220 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57667629020934 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32714896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12840015 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64427156263936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64427131491328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32736536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59216410763264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32714896 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚66 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18226314 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56880673646592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 789192483328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚849 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14510222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4327448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 % _timestamp‚1.7884410112612815e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58426435908096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 789974855168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18226314 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 786955374342 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12839483 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184415232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65256387467776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14510220 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18204676 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 829255976448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58426435908096 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 789192483328B;ÙŸ[0Ù+ _wandbruntime‚728  _runtime‚728#úôALÚÆ /loss/FRt2/assignment_loss‚7.316902902391222 /loss/FRt4/assignment_loss‚7.377922065922471 0loss/FBt3/assignment_loss‚0.5417080752996563 .loss/FRt4/detection_loss‚2.372752602105258 trainer/global_step‚849 _runtime‚ 728.3101212 1loss/SRqqt1/assignment_loss‚2.739732618455763 2loss/SRqqt2/assignment_loss‚2.1292729718344554 .loss/FBt2/detection_loss‚4.583427255352338 % _timestamp‚1.7884410112630162e+09 1loss/FBt4/assignment_loss‚0.29827232864259245 /loss/FRt1/detection_loss‚2.3641510731058903 /loss/FRt1/assignment_loss‚7.211790602782677 1loss/SRqqt3/assignment_loss‚3.388581390734072 2loss/SRqqt4/assignment_loss‚2.4490333863946256 /loss/FRt3/detection_loss‚3.5644247122108936 1loss/SRqqt1/detection_loss‚1.1749914095363392 0loss/SRqqt3/detection_loss‚4.979094366232554 1loss/SRqqt4/detection_loss‚2.0098637850749017 .loss/FBt1/detection_loss‚3.286380103656224 .loss/FBt3/detection_loss‚3.544787072226402 0loss/FBt1/assignment_loss‚0.6088364149005169 epoch‚0 _step‚67 .loss/FRt2/detection_loss‚3.797386944293976 .loss/FRt3/assignment_loss‚7.32424521446228 0loss/FBt2/assignment_loss‚0.3092009531523824 0loss/SRqqt2/detection_loss‚4.146423636163984 .loss/FBt4/detection_loss‚5.203484268353854 %loss/total_loss‚82.72266615328533è(®PÛÊ _step‚67 .loss/FRt2/detection_loss‚3.797386944293976 .loss/FRt3/assignment_loss‚7.32424521446228 0loss/FBt2/assignment_loss‚0.3092009531523824 0loss/SRqqt2/detection_loss‚4.146423636163984 .loss/FBt4/detection_loss‚5.203484268353854 %loss/total_loss‚82.72266615328533 /loss/FRt2/assignment_loss‚7.316902902391222 /loss/FRt4/assignment_loss‚7.377922065922471 0loss/FBt3/assignment_loss‚0.5417080752996563 .loss/FRt4/detection_loss‚2.372752602105258 trainer/global_step‚849 _runtime‚ 728.3101212 1loss/SRqqt1/assignment_loss‚2.739732618455763 2loss/SRqqt2/assignment_loss‚2.1292729718344554 .loss/FBt2/detection_loss‚4.583427255352338 % _timestamp‚1.7884410112630162e+09 1loss/FBt4/assignment_loss‚0.29827232864259245 /loss/FRt1/detection_loss‚2.3641510731058903 /loss/FRt1/assignment_loss‚7.211790602782677 1loss/SRqqt3/assignment_loss‚3.388581390734072 2loss/SRqqt4/assignment_loss‚2.4490333863946256 /loss/FRt3/detection_loss‚3.5644247122108936 1loss/SRqqt1/detection_loss‚1.1749914095363392 0loss/SRqqt3/detection_loss‚4.979094366232554 1loss/SRqqt4/detection_loss‚2.0098637850749017 .loss/FBt1/detection_loss‚3.286380103656224 .loss/FBt3/detection_loss‚3.544787072226402 0loss/FBt1/assignment_loss‚0.6088364149005169 epoch‚0C‚ùÐ3\Ü‚(:ˆ ¹ãåÔ䜬Ç gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚™tÎü714.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚189.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c—ýCÝ‚(:¸ ¹ãåÔÚÔ™Ê disk.drbd2080.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 26.5234375 disk.sda3.in‚0 cpu‚25.045960831528387 disk.sda.in‚0! proc.memory.rssMB‚ 2629.828125* proc.memory.percent‚16.051197052001953" disk./.usageGB‚289.90771484375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.9833984375 disk.sda1.in‚0 network.sent‚ 1.09161e+06* disk./.usagePercent‚19.455949991938834 disk.drbd1716.out‚0$ disk.nvme0n1.out‚2369.4443359375 disk.sda2.in‚0$ memory_percent‚95.12701034545898 disk.dm-1.out‚ 26.5234375 disk.dm-0.in‚0.078125 disk.sda1.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 798.390625 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0c0n1.out‚2369.4443359375 network.recv‚334075 disk.drbd1716.in‚0! disk.dm-0.out‚2369.4443359375 disk.rbd0.out‚0 disk.sda3.out‚ 26.5234375 disk.rbd1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0—¥HžÞ‚(:“ ÈãåÔÅߤÇ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FS¯íKß‚(:À ÈãåÔªë§Ê disk.sda1.out‚0 disk.nvme0n1.in‚0.078125# proc.memory.rssMB‚ 2629.83203125 disk.dm-1.in‚0 disk.dm-1.out‚ 27.01953125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 27.01953125$ disk.nvme0n1.out‚2420.5693359375 network.recv‚338912! disk.dm-0.out‚2420.5693359375" memory_percent‚95.127272605896 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 798.34765625 network.sent‚ 1.140378e+06$ disk./.usageGB‚289.9423599243164 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2420.5693359375 disk.drbd1716.out‚0 cpu‚25.046031070425062 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.rbd1.out‚ 5.0458984375 disk.sda.out‚ 27.01953125 disk.drbd1716.in‚0* proc.memory.percent‚16.051220893859863 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.sda3.in‚0* disk./.usagePercent‚19.458275052364804 disk.drbd2266.in‚0Ž t“à‚(:ˆ ×ãåÔЊ™Ç gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚194.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.8094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýd­Á@á‚(:µ ×ãåÔó‚Ê disk.dm-1.in‚0 disk.drbd1716.in‚0* disk./.usagePercent‚19.459030786605883 disk.drbd2266.out‚0 network.sent‚ 1.146188e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚27.625 disk.nvme0n1.in‚0.078125 disk.rbd1.out‚ 5.0458984375 cpu‚25.045802876274365 disk.sda.in‚0 disk.sda.out‚27.625 disk.sda3.in‚0% disk.nvme0n1.out‚2478.78857421875 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚342951 disk.rbd0.out‚0$ memory_percent‚95.12581825256348 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚27.625" disk.dm-0.out‚2478.78857421875 disk.drbd2266.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚2478.78857421875' proc.memory.availableMB‚ 798.5859375" proc.memory.rssMB‚ 2629.8359375% disk./.usageGB‚289.95362091064453 disk.sda2.in‚0* proc.memory.percent‚16.051244735717773’ö4ô0â+ _wandbruntime‚769  _runtime‚769. mmÃã½ ' lr-AdamW/pg1‚0.00011454265883412437 ' lr-AdamW/pg2‚0.00011454265883412437 trainer/global_step‚899 # _timestamp‚1.78844105293013e+09 _runtime‚ 769.97733342 _step‚68[9·ÇäÁ ' lr-AdamW/pg2‚0.00011454265883412437 trainer/global_step‚899 # _timestamp‚1.78844105293013e+09 _runtime‚ 769.97733342 _step‚68 ' lr-AdamW/pg1‚0.00011454265883412437DèÀ*10å+ _wandbruntime‚769  _runtime‚7698‚ÂÏ'æÉO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 834653683712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15345180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184428544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15345180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61789383009792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4578296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62624819052032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62624019523072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19253394 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62624019523072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34620188 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68135558817792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 877131729408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60154728264704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4577771 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15345182 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19275006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60986988133202 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚ 769.978241704 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61789365839360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61789365839360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209201152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69012899748352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1015 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68135583590400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 877316157952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 834653683712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13579943 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62624819052032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9002172 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69012690547200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13580471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19275006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚899 _step‚69 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832242829138 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835436042240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835436042240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 831461635922 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34598574 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61789383009792q ‰Ó'çÍO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19253394 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62624019523072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34620188 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68135558817792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 877131729408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60154728264704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4577771 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15345182 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19275006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60986988133202 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚ 769.978241704 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61789365839360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61789365839360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209201152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69012899748352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1015 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1035 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68135583590400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 877316157952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 834653683712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13579943 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62624819052032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9002172 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69012690547200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13580471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19275006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚899 _step‚69 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832242829138 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835436042240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835436042240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 831461635922 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34598574 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61789383009792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 834653683712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15345180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184428544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15345180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61789383009792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4578296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62624819052032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62624019523072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592EŒ˜0è+ _wandbruntime‚770  _runtime‚770sFFl¬éÓM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13595062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4582898 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19296366 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182318080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 832385592326 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62694378124288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69089570841600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60221546053632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13595594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61857997412864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34658588 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68211265780736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9012167 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15362220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835581169152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884410537246716e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19296366 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512+šÝ&ê×M OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13595594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61857997412864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34658588 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68211265780736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9012167 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15362220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835581169152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884410537246716e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19296366 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13595062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4582898 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19296366 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182318080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 832385592326 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62694378124288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69089570841600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60221546053632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512F#…„»0ë+ _wandbruntime‚770  _runtime‚770Љç×PìÊ trainer/global_step‚899 1loss/FBt2/assignment_loss‚0.39003861272351514 1loss/SRqqt1/detection_loss‚1.2261016983966082 1loss/SRqqt4/detection_loss‚2.7507203894022565 /loss/FBt2/detection_loss‚3.3033470161259175 /loss/FBt3/detection_loss‚0.8865982670221499 _runtime‚ 770.773758529 1loss/FBt1/assignment_loss‚0.32686475107816065 0loss/FBt3/assignment_loss‚0.5887979452430316 0loss/SRqqt2/detection_loss‚5.607160707314809 0loss/SRqqt3/detection_loss‚3.170242492179386 %loss/total_loss‚80.04123117926578 $ _timestamp‚1.788441053726453e+09 /loss/FRt3/assignment_loss‚7.346855099995931 /loss/FRt4/assignment_loss‚7.334616352961613 -loss/FBt1/detection_loss‚5.63825825703415 _step‚71 0loss/SRqqt1/assignment_loss‚2.58199278728382 /loss/FRt1/detection_loss‚2.2875627167522907 epoch‚0 2loss/SRqqt2/assignment_loss‚2.8376289159059525 2loss/SRqqt3/assignment_loss‚2.9444366455078126 1loss/SRqqt4/assignment_loss‚2.520199028345255 /loss/FRt4/detection_loss‚2.2383210969658998 /loss/FRt1/assignment_loss‚7.235932479470463 /loss/FRt2/assignment_loss‚7.257817857405719 /loss/FBt4/detection_loss‚3.6502542730963463 /loss/FRt2/detection_loss‚3.3674793664146874 /loss/FBt4/assignment_loss‚0.083580092564425 .loss/FRt3/detection_loss‚4.466424330075582šßYTíÎ /loss/FRt1/assignment_loss‚7.235932479470463 /loss/FRt2/assignment_loss‚7.257817857405719 /loss/FBt4/detection_loss‚3.6502542730963463 /loss/FRt2/detection_loss‚3.3674793664146874 /loss/FBt4/assignment_loss‚0.083580092564425 .loss/FRt3/detection_loss‚4.466424330075582 trainer/global_step‚899 1loss/FBt2/assignment_loss‚0.39003861272351514 1loss/SRqqt1/detection_loss‚1.2261016983966082 1loss/SRqqt4/detection_loss‚2.7507203894022565 /loss/FBt2/detection_loss‚3.3033470161259175 /loss/FBt3/detection_loss‚0.8865982670221499 _runtime‚ 770.773758529 1loss/FBt1/assignment_loss‚0.32686475107816065 0loss/FBt3/assignment_loss‚0.5887979452430316 0loss/SRqqt2/detection_loss‚5.607160707314809 0loss/SRqqt3/detection_loss‚3.170242492179386 %loss/total_loss‚80.04123117926578 $ _timestamp‚1.788441053726453e+09 /loss/FRt3/assignment_loss‚7.346855099995931 /loss/FRt4/assignment_loss‚7.334616352961613 -loss/FBt1/detection_loss‚5.63825825703415 _step‚71 0loss/SRqqt1/assignment_loss‚2.58199278728382 /loss/FRt1/detection_loss‚2.2875627167522907 epoch‚0 2loss/SRqqt2/assignment_loss‚2.8376289159059525 2loss/SRqqt3/assignment_loss‚2.9444366455078126 1loss/SRqqt4/assignment_loss‚2.520199028345255 /loss/FRt4/detection_loss‚2.2383210969658998GÏ}e…“î‚(:ˆ æãåÔÒÙÞÇ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚189.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.9446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*íFï‚(:» æãåÔ³ùÂÉ disk.rbd0.in‚0% disk.nvme0n1.out‚2529.55419921875" proc.memory.rssMB‚ 2629.8359375* proc.memory.percent‚16.051244735717773% disk./.usageGB‚289.90278244018555 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 28.0546875 disk.sda3.in‚0 disk.dm-1.out‚ 28.0546875' proc.memory.availableMB‚ 798.5234375 network.sent‚ 1.152496e+06 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚2529.55419921875* disk./.usagePercent‚19.455618974197062 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 28.0546875 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125$ memory_percent‚95.12619972229004 cpu‚25.045919099904072 disk.drbd1716.out‚0" disk.dm-0.out‚2529.55419921875 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚347738 disk.drbd2080.in‚0 disk.rbd1.out‚5.1875¤ ¤Åžð‚(:“ õãåÔ”ËÒÇ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.701' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.740199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôB6à:ñ‚(:¯ õãåÔ×ùÞÉ" proc.memory.rssMB‚ 2629.8359375 network.recv‚352060* disk./.usagePercent‚19.458461425849734 disk.dm-1.out‚ 28.796875 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚2590.64404296875" disk.dm-0.out‚2590.64404296875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2590.64404296875 disk.sda.out‚ 28.796875! proc.memory.availableMB‚798.5 disk.rbd1.in‚0* proc.memory.percent‚16.051244735717773 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.201419e+06 disk.drbd2265.in‚0 cpu‚25.04597954577766$ disk./.usageGB‚289.9451370239258 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚5.1875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0# memory_percent‚95.1263427734375 disk.sda3.out‚ 28.796875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125˜ƒ²“ò‚(:ˆ „äåÔ­ä•Ç gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚195.716' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.1432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âéý6ó‚(:« „äåÔ°¤£É disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚29.09375! proc.memory.availableMB‚798.5* proc.memory.percent‚16.051244735717773 network.sent‚ 1.20739e+06% disk./.usageGB‚289.96062088012695% disk.nvme0n1.out‚2636.06591796875 disk.dm-0.in‚0.078125" disk.dm-0.out‚2636.06591796875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚19.459500560431213 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda.out‚29.09375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚5.25 disk.sda3.in‚0# memory_percent‚95.1263427734375 proc.cpu.threads‚17 network.recv‚356429 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2629.8359375' disk.nvme0c0n1.out‚2636.06591796875 disk.sda3.out‚29.09375 cpu‚25.045782020582294{Ø­0ô+ _wandbruntime‚812  _runtime‚812uñ5ÔÆõÀ _step‚72 ' lr-AdamW/pg1‚0.00012091321827984874 ' lr-AdamW/pg2‚0.00012091321827984874 trainer/global_step‚949 % _timestamp‚1.7884410954598408e+09 _runtime‚ 812.507071893õš ÊöÄ trainer/global_step‚949 % _timestamp‚1.7884410954598408e+09 _runtime‚ 812.507071893 _step‚72 ' lr-AdamW/pg1‚0.00012091321827984874 ' lr-AdamW/pg2‚0.00012091321827984874HÔÅ”0÷+ _wandbruntime‚812  _runtime‚812:/K¡Î'øÈO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16197180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 926354300928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9501775 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186525696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20323446 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36542240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65220944514560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71919668334592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877673047330 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9501772 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65220961684992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20345058 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66101986884096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211298304 $ _timestamp‚1.788441095460547e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16197182 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20323446 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 881824728064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66102786413056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72845836109824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16197180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66101986884096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65220961684992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 881042369536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36542240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65220944514560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71919668334592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877673047330 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9501772 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65220961684992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20345058 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66101986884096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211298304 $ _timestamp‚1.788441095460547e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16197182 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20323446 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 881824728064 H>DeviceStatsMonitor.f4»_on_train_batch_start/allocation.all.current‚21614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66102786413056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72845836109824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16197180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66101986884096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65220961684992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 881042369536 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 926354300928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9501775 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186525696I·\0ú+ _wandbruntime‚813  _runtime‚813™˜¤îÙ&ûÓM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20344780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚949 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66172345485312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36559000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 % _timestamp‚1.7884410962671266e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63562452539392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚74 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65289576088064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64441032503766 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20366418 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200799232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 927171827200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64441830736550 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 879378197158 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14351542 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 927347853824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4840305 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20366418 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36580640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66171545943040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176026624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65289593258496 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 881969854976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1037 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65289576088064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 882752226816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4839778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63562435500032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16214220 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20344780 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16214222 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9511767 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72922723151360 _runtime‚ 813.314912919 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9511764 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36580640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16214220 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65289593258496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71995375297536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72922522352128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36559000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592L¯tÝ&ü×M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20344780 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16214222 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9511767 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72922723151360 _runtime‚ 813.314912919 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9511764 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36580640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16214220 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65289593258496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71995375297536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72922522352128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36559000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20344780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚949 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66172345485312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36559000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 % _timestamp‚1.7884410962671266e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63562452539392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚74 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65289576088064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64441032503766 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20366418 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200799232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 927171827200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64441830736550 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 879378197158 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14351542 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 927347853824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4840305 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20366418 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36580640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66171545943040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176026624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65289593258496 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 881969854976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1037 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65289576088064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 882752226816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4839778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63562435500032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16214220 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512Jz4©™0ý+ _wandbruntime‚813  _runtime‚813&ÇãSþÍ 0loss/SRqqt2/detection_loss‚4.388198427085219 .loss/FBt3/detection_loss‚6.594336993992381 %loss/total_loss‚81.04332189946766 2loss/SRqqt3/assignment_loss‚3.0027873263214575 /loss/FBt2/detection_loss‚3.3071209341287613 .loss/FBt4/detection_loss‚3.140873029332154 _step‚75 0loss/FBt1/assignment_loss‚0.5158017223570823 1loss/SRqqt3/detection_loss‚2.3007368291438044 % _timestamp‚1.7884410962688956e+09 /loss/FRt1/assignment_loss‚7.131152540842692 1loss/FBt2/assignment_loss‚0.28878128815753173 /loss/FRt4/assignment_loss‚7.382280341365881 2loss/SRqqt2/assignment_loss‚2.8521077386264144 /loss/FRt2/detection_loss‚4.5586495995521545 trainer/global_step‚949 0loss/FRt2/assignment_loss‚7.1775519507271905 .loss/FRt3/assignment_loss‚7.42966349919637 2loss/SRqqt1/assignment_loss‚2.7315824188684164 0loss/FBt4/assignment_loss‚0.6571902771910542 /loss/FRt4/detection_loss‚2.5274223148038515 0loss/SRqqt4/detection_loss‚1.780323451022721 _runtime‚ 813.315997726 2loss/SRqqt4/assignment_loss‚2.6243394282129078 0loss/FBt3/assignment_loss‚0.3469301239149445 /loss/FRt3/detection_loss‚3.6777980675299964 0loss/SRqqt1/detection_loss‚1.298952104043409 epoch‚0 /loss/FRt1/detection_loss‚1.6267788078387577 .loss/FBt1/detection_loss‚3.701962685212493"¯ %WÿÑ 0loss/SRqqt2/detection_loss‚4.388198427085219 .loss/FBt3/detection_loss‚6.594336993992381 %loss/total_loss‚81.04332189946766 2loss/SRqqt3/assignment_loss‚3.0027873263214575 /loss/FBt2/detection_loss‚3.3071209341287613 .loss/FBt4/detection_loss‚3.140873029332154 _step‚75 0loss/FBt1/assignment_loss‚0.5158017223570823 1loss/SRqqt3/detection_loss‚2.3007368291438044 % _timestamp‚1.7884410962688956e+09 /loss/FRt1/assignment_loss‚7.131152540842692 1loss/FBt2/assignment_loss‚0.28878128815753173 /loss/FRt4/assignment_loss‚7.382280341365881 2loss/SRqqt2/assignment_loss‚2.8521077386264144 /loss/FRt2/detection_loss‚4.5586495995521545 trainer/global_step‚949 0loss/FRt2/assignment_loss‚7.1775519507271905 .loss/FRt3/assignment_loss‚7.42966349919637 2loss/SRqqt1/assignment_loss‚2.7315824188684164 0loss/FBt4/assignment_loss‚0.6571902771910542 /loss/FRt4/detection_loss‚2.5274223148038515 0loss/SRqqt4/detection_loss‚1.780323451022721 _runtime‚ 813.315997726 2loss/SRqqt4/assignment_loss‚2.6243394282129078 0loss/FBt3/assignment_loss‚0.3469301239149445 /loss/FRt3/detection_loss‚3.6777980675299964 0loss/SRqqt1/detection_loss‚1.298952104043409 epoch‚0 /loss/FRt1/detection_loss‚1.6267788078387577 .loss/FBt1/detection_loss‚3.701962685212493Kú*ö“€‚(:ˆ “äåÔ”„ÉÇ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.307' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªñò8‚(:­ “äåÔÛžÒÉ disk.drbd1716.out‚0 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚2731.39306640625$ memory_percent‚95.12670040130615* disk./.usagePercent‚19.463319424819474$ disk./.usageGB‚290.0175247192383 disk.dm-1.out‚29.625 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda3.out‚29.625* proc.memory.percent‚16.051244735717773 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2629.8359375 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚2731.39306640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚29.625( proc.memory.availableMB‚ 798.44140625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.046127913665043 proc.cpu.threads‚17 network.recv‚361065 disk.dm-1.in‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚2731.39306640625 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 network.sent‚ 1.213791e+06 disk.drbd2266.out‚0 disk.rbd1.out‚5.25¯ý˜“‚‚(:ˆ ¢äåÔŽ¤Ç gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TˆÍ16ƒ‚(:¿ ¢äåÔõ¡ŒÊ disk.sda.out‚ 30.1484375K|·ƒ disk.sda2.out‚0 disk.dm-1.out‚ 30.1484375 cpu‚25.045879633655133 disk.sda3.out‚ 30.1484375 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.12827396392822 disk.rbd1.out‚ 5.4775390625 network.sent‚ 1.2639e+06 disk.drbd2265.in‚0' disk.nvme0c0n1.out‚2805.13916015625 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2629.8359375* proc.memory.percent‚16.051244735717773 disk.rbd1.in‚0 network.recv‚366800* disk./.usagePercent‚19.464382624109174$ disk./.usageGB‚290.0333671569824 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-0.in‚0.078125( proc.memory.availableMB‚ 798.18359375 proc.cpu.threads‚17 disk.drbd2080.in‚0" disk.dm-0.out‚2805.13916015625 disk.nvme0c0n1.in‚0.078125 disk.nvme0n1.in‚0.078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚2805.13916015625 disk.rbd0.in‚0{(tž„‚(:“ ±äåÔïë¨Ç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚132.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.518200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01vXH…‚(:½ ±äåÔõê¬É disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 cpu‚25.045993056667513 network.sent‚ 1.269154e+06 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚2858.20751953125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 30.7421875 disk.dm-1.in‚0 network.recv‚370439( disk./.usagePercent‚19.4658003938338 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125% proc.memory.availableMB‚ 798.40625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 30.7421875$ memory_percent‚95.12691497802734 proc.cpu.threads‚17% disk./.usageGB‚290.05449295043945' disk.nvme0c0n1.out‚2858.20751953125 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125* proc.memory.percent‚16.051244735717773 disk.sda.in‚0 disk.sda.out‚ 30.7421875 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚2858.20751953125" proc.memory.rssMB‚ 2629.8359375 disk.drbd2080.out‚0 disk.rbd1.out‚ 5.6572265625ü:0†+ _wandbruntime‚855  _runtime‚855‰ä‹7ƇÀ ' lr-AdamW/pg2‚0.00012728377772557313 trainer/global_step‚999 % _timestamp‚1.7884411379955316e+09 _runtime‚ 855.042757726 _step‚76 ' lr-AdamW/pg1‚0.00012728377772557313ûMm0ÊˆÄ _runtime‚ 855.042757726 _step‚76 ' lr-AdamW/pg1‚0.00012728377772557313 ' lr-AdamW/pg2‚0.00012728377772557313 trainer/global_step‚999 % _timestamp‚1.7884411379955316e+09L,[3!0‰+ _wandbruntime‚855  _runtime‚855 †ÒpÑ'ŠËO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 924665651954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21415110 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75703777851392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69580753774080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5091499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 928213413888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21393498 % _timestamp‚1.7884411379962568e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67761189848818 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17049180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38442678 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69579954245120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10001375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76679215656448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68652540360192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66836507157504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 _step‚77 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15092871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21415110 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75703802624000 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38464292 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67761189848818 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17049180 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38442678 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69579954245120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10001375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76679215656448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68652540360192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190720000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66836507157504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 _step‚77 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15092871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215492608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21415110 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75703802624000 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38464292 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 924665651954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21415110 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75703777851392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69580753774080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5091499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Mœ{ß60Œ+ _wandbruntime‚855  _runtime‚855‰­ó#ÕM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68721154763264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69650312846336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69649513304064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 925589608566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15107951 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76755876071424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75779484814336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69650312846336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38502692 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10011367 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17066220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21436470 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67828931594358 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5097126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217576448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 924808415142 B9DeviceStatsMonitû,‚¯èor.on_train_batch_end/segment.all.current‚1010 _step‚78 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17066220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21436470 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67828931594358 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5097126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217576448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 924808415142 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 _step‚78 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68721154763264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69650312846336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69649513304064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 925589608566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15107951 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76755876071424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75779484814336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69650312846336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38502692 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10011367 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5347416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213395456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22463550 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72084101864960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 973819741184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72084119035392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73057921606144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 970877063362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10500972 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0,· îv›ÒO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22485162 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1049 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 974602099712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15847854 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17901180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10500975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80512172700672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79487912140800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70177413643264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72084101864960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71148290706626 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5346882 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17901180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40386344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22485162 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5347416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213395456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22463550 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72084101864960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 973819741184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72084119035392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73057921606144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 970877063362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10500972 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on¢|R­b_train_batch_start/segment.large_pool.peak‚521 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73058721135104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 974602099712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 % _timestamp‚1.7884411803968978e+09 _step‚81 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71147492474050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79487887368192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70177396603904QìLã!0œ+ _wandbruntime‚898  _runtime‚898´™NHÞ&ØM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79563594331136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10510967 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 975529598464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22506522 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72152750608896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5352483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚1049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22506522 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 974747226624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5351948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70244214392832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15863450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40403104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72152733438464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72152750608896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40403104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71216032452166 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17918222 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 975529598464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17918220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22484884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1025053381120 _runtime‚ 898.246141405 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80588836321792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 974747226624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73128280207360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17918220 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40424744 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71215234219382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80588622939648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79563569558528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22484884 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 971801019974 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚82 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10510964 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15862912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17918222 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10510967 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 975529598464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22506522 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72152750608896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5352483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚1049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22506522 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 974747226624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5351948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70244214392832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15863450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40403104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72152733438464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72152750608896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40403104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71216032452166 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17918222 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 975529598464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17918220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22484884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1025053381120 _runtime‚ 898.246141405 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80588836321792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 974747226624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73128280207360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17918220 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40424744 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71215234219382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80588622939648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79563569558528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22484884 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 971801019974 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚82 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10510964 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15862912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17918222 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16604466 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1020208427008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42286782 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18753182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73518303089664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚ 939.70931787 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84345247270912 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84345040166912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1020208427008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1073225613312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚1099 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75515680540160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23533602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1017088474770 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76535888967168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23555214 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1016307281554 % _timestamp‚1.7884412226618202e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76536688496128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18753182 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603348 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42308396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74535391564434 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74534593331858 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1073043281920ï$éÝ'­×O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76536688496128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18753182 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603348 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42308396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74535391564434 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74534593331858 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1073043281920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1020990785536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11000575 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚85 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23533602 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76536688496128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18753180 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73518286050304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16604466 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1020208427008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42286782 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18753182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182331392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73518303089664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚ 939.70931787 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207104000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84345247270912 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84345040166912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1020208427008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1073225613312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚1099 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75515680540160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23533602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798232576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1017088474770 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76535888967168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 782358528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23555214 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1016307281554 % _timestamp‚1.7884412226618202e+09Uç¥â 0®+ _wandbruntime‚940  _runtime‚940Z¼?nä&¯ÞM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16619514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5608947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23554936 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16618974 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74602335077190 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75584329284096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884412234543355e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1017231237958 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42346796 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76605448026112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74603133309974 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83347703847936 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18770222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18770220 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83347679075328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1018012431382 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42325156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23576574 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1021135912448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11010564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 940.502243759 _step‚86 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1074211102208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42346796 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1021918284288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42325156 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1021918284288 trainer/global_step‚1099 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1074022492672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76606247568384çˆï °âM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18770222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18770220 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83347679075328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1018012431382 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42325156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23576574 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1021135912448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11010564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 940.502243759 _step‚86 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1074211102208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42346796 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1021918284288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42325156 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1021918284288 trainer/global_step‚1099 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1074022492672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76606247568384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16619514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5608947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23554936 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16618974 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74602335077190 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75584329284096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884412234543355e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1017231237958 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42346796 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76605448026112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74603133309974 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83347703847936 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2V"mó0±+ _wandbruntime‚940  _runtime‚940ª-™M²Ç 0loss/SRqqt2/detection_loss‚3.492369122082187 _runtime‚ 940.50331467 _step‚87 trainer/global_step‚1099 epoch‚0 0loss/FRt3/assignment_loss‚6.6327999738546515 /loss/FRt2/detection_loss‚2.9970787465572357 0loss/SRqqt1/detection_loss‚2.109114761019487 /loss/SRqqt3/detection_loss‚4.09496267254536 /loss/FRt1/assignment_loss‚5.438219524198963 1loss/SRqqt3/assignment_loss‚3.053160630739652 -loss/FRt1/detection_loss‚2.37923597424261 .loss/FRt4/detection_loss‚2.396817231764559 .loss/FBt3/detection_loss‚4.941204946275794 .loss/FRt2/assignment_loss‚6.59664249420166 0loss/FRt4/assignment_loss‚4.7721115174840705 0loss/FBt3/assignment_loss‚0.5981038422323763 % _timestamp‚1.7884412234562073e+09 2loss/SRqqt2/assignment_loss‚2.5359263857526164 2loss/SRqqt4/assignment_loss‚2.6225948553336296 1loss/FBt2/assignment_loss‚0.35262708960024236 -loss/FBt1/detection_loss‚4.30948727329572 2loss/SRqqt1/assignment_loss‚1.9218924052929922 1loss/SRqqt4/detection_loss‚1.7666694646407115 /loss/FBt2/detection_loss‚4.2889909913827635 /loss/FBt4/detection_loss‚2.9973212088558974 %loss/total_loss‚75.44391565275559 /loss/FBt1/assignment_loss‚0.683850770413604 0loss/FBt4/assignment_loss‚0.3830046437926487 .loss/FRt3/detection_loss‚4.079729127196165” <\Q³Ë epoch‚0 0loss/FRt3/assignment_loss‚6.6327999738546515 /loss/FRt2/detection_loss‚2.9970787465572357 0loss/SRqqt1/detection_loss‚2.109114761019487 /loss/SRqqt3/detection_loss‚4.09496267254536 /loss/FRt1/assignment_loss‚5.438219524198963 1loss/SRqqt3/assignment_loss‚3.053160630739652 -loss/FRt1/detection_loss‚2.37923597424261 .loss/FRt4/detection_loss‚2.396817231764559 .loss/FBt3/detection_loss‚4.941204946275794 .loss/FRt2/assignment_loss‚6.59664249420166 0loss/FRt4/assignment_loss‚4.7721115174840705 0loss/FBt3/assignment_loss‚0.5981038422323763 % _timestamp‚1.7884412234562073e+09 2loss/SRqqt2/assignment_loss‚2.5359263857526164 2loss/SRqqt4/assignment_loss‚2.6225948553336296 1loss/FBt2/assignment_loss‚0.35262708960024236 -loss/FBt1/detection_loss‚4.30948727329572 2loss/SRqqt1/assignment_loss‚1.9218924052929922 1loss/SRqqt4/detection_loss‚1.7666694646407115 /loss/FBt2/detection_loss‚4.2889909913827635 /loss/FBt4/detection_loss‚2.9973212088558974 %loss/total_loss‚75.44391565275559 /loss/FBt1/assignment_loss‚0.683850770413604 0loss/FBt4/assignment_loss‚0.3830046437926487 .loss/FRt3/detection_loss‚4.079729127196165 0loss/SRqqt2/detection_loss‚3.492369122082187 _runtime‚ 940.50331467 _step‚87 trainer/global_step‚1099WeªŽÆ´‚(:’ ‹ååÔ×áÁÇ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚197.06' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.412000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡GŠIDµ‚(:¹ ‹ååÔ•“ÅÉ disk.drbd2080.in‚0 disk.drbd1716.in‚0$ memory_percent‚95.12832164764404 disk.sda1.out‚0& disk.nvme0c0n1.out‚3191.9130859375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 34.078125* disk./.usagePercent‚19.466870505326924 disk.nvme0n1.in‚0.078125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 34.078125* proc.memory.percent‚16.051292419433594 network.recv‚397532% disk./.usageGB‚290.07043838500977 disk.rbd1.out‚ 5.923828125 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 34.078125 disk.dm-1.in‚0 cpu‚25.046096568707256$ disk.nvme0n1.out‚3191.9130859375 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 798.17578125 proc.memory.rssMB‚ 2629.84375 proc.cpu.threads‚17 network.sent‚ 1.391856e+06! disk.dm-0.out‚3191.9130859375Y]ž¶‚(:“ šååÔÊÔãÇ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.771' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.154199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¶2!>·‚(:³ šååÔù¨´É disk.drbd2266.out‚0 disk.rbd1.in‚0 cpu‚25.04616302781773 network.recv‚403466 disk.dm-1.out‚34.5625$ disk.nvme0n1.out‚3240.7080078125 disk.drbd2080.out‚0 proc.memory.rssMB‚ 2629.84375 disk.drbd2265.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0! disk.dm-0.out‚3240.7080078125( proc.memory.availableMB‚ 798.09765625* proc.memory.percent‚16.051292419433594* disk./.usagePercent‚19.468181519909766$ memory_percent‚95.12879848480225 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.442083e+06% disk./.usageGB‚290.08997344970703 disk.sda.out‚34.5625 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.out‚ 6.0654296875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚34.5625 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3240.7080078125uã1vG¸‚(:¼ ©ååÔü¯ŒË disk.sda2.out‚0$ memory_percent‚95.13037204742432 cpu‚25.046132092410595 disk.dm-1.out‚ 35.0859375$ disk.nvme0n1.out‚3296.1142578125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda.out‚ 35.0859375 disk.sda3.in‚0* proc.memory.percent‚16.051292419433594 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚290.1044387817383 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda2.in‚0* disk./.usagePercent‚19.469152300479607 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 network.sent‚ 1.447322e+06! disk.dm-0.out‚3296.1142578125& disk.nvme0c0n1.out‚3296.1142578125 proc.cpu.threads‚17 network.recv‚406886 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 35.0859375( proc.memory.availableMB‚ 797.83984375 proc.memory.rssMB‚ 2629.84375 disk.drbd2266.in‚0 disk.rbd1.out‚ 6.0654296875 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0I÷H¡“¹‚(:ˆ ©ååÔí«Ô× gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.078' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.4156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å4Š0º+ _wandbruntime‚981  _runtime‚981 þÒÇ»Á ' lr-AdamW/pg1‚0.00014639545606274625 ' lr-AdamW/pg2‚0.00014639545606274625 trainer/global_step‚1149 % _timestamp‚1.7884412648726454e+09 _runtime‚ 981.919848046 _step‚88Rª‚'˼Štrainer/global_step‚1149 % _timestamp‚1.7884412648726454e+09 _runtime‚ 981.919848046 _step‚88 ' lr-AdamW/pg1‚0.00014639545606274625 ' lr-AdamW/pg2‚0.00014639545606274625XtÓ*O0½+ _wandbruntime‚981  _runtime‚981FNs¡Ú'¾ÔO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76859175496704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5859775 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87056106401792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1062518692962 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24603654 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1067379471360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88178104251904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1121997850112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80014655857152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78947259215360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11500172 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88178326035968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚1149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76859192536064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80014655857152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80013856328192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1066597112832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1066597112832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24625266 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1067379471360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17359950 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17359406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5859234 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 _step‚89 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19605180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77922492422242 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1063299886178 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚ 981.920760488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24603654 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44230448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80013856328192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11500175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚789472763857926Ï„Pf¿ØO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 _step‚89 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12012354048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19605180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221784064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77922492422242 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1063299886178 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚ 981.920760488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24603654 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 799528960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44230448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80013856328192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 782358528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11500175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78947276385792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76859175496704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5859775 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87056106401792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1062518692962 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24603654 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019218389 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1067379471360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88178104251904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197011456 A8DeviceStatsMonitor.on_train_batch_start/num_devic{K„.x e_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1121997850112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80014655857152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78947259215360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11500172 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88178326035968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚1149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76859192536064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80014655857152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80013856328192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1066597112832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1066597112832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24625266 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1067379471360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635023317 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17359950 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17359406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5859234 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Y,Kx0À+ _wandbruntime‚982  _runtime‚982çÚÏä&ÁÞM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44247208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76925993285632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1123165073408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24646626 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79015890788864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1067524598272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77990234167782 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76926010324992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19622222 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87131788592128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17375037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11510164 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1063442649366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1068306970112 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80084214929408 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88254978438144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17374521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚1149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24646626 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79015907959296 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80084214929408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19622222 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80083415387136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88254765056000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44268848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44268848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79015890788864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1067524598272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79015907959296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1068306970112 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77989435934998 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _runtime‚ 982.719284975 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1122976463872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11510167 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0€$rè&ÂâM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19622222 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87131788592128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17375037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11510164 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1063442649366 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1068306970112 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80084214929408 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88254978438144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17374521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚1149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24646626 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79015907959296 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80084214929408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19622222 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80083415387136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88254765056000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44268848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44268848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79015890788864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1067524598272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79015907959296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1068306970112 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77989435934998 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _runtime‚ 982.719284975 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12012354048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1122976463872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11510167 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44247208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76925993285632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1123165073408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24646626 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 782371840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 799542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79015890788864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1067524598272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77990234167782 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76926010324992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Z¹1ÀÃ0Ã+ _wandbruntime‚982  _runtime‚982ßÊNÄÈ .loss/FRt4/detection_loss‚2.547754468159242 $ _timestamp‚1.788441265673263e+09 /loss/FRt2/assignment_loss‚6.044729483755011 0loss/FBt3/assignment_loss‚0.7342745287266249 .loss/FRt1/detection_loss‚2.429312794390371 /loss/FRt3/detection_loss‚3.0026081284961186 /loss/SRqqt3/detection_loss‚2.79802034747216 /loss/FBt2/detection_loss‚2.0442936741992046 /loss/FBt4/detection_loss‚2.5070992298343455 epoch‚0 /loss/FRt1/assignment_loss‚5.684199502912619 1loss/SRqqt2/assignment_loss‚2.771911434084177 1loss/SRqqt1/detection_loss‚1.8776872882412539 /loss/FBt1/assignment_loss‚0.575081705847489 0loss/FBt2/assignment_loss‚0.3657507167982706 trainer/global_step‚1149 _step‚91 /loss/FRt3/assignment_loss‚6.262602239041715 /loss/FRt4/assignment_loss‚5.081446838378906 1loss/SRqqt4/assignment_loss‚2.809407435275696 0loss/SRqqt2/detection_loss‚4.496247085928917 2loss/SRqqt3/assignment_loss‚3.3675556336679766 _runtime‚ 982.720380663 2loss/SRqqt1/assignment_loss‚2.0076165827405124 .loss/FRt2/detection_loss‚2.571708792134335 1loss/SRqqt4/detection_loss‚1.4684118766869818 %loss/total_loss‚68.65963073121007 /loss/FBt1/detection_loss‚4.8372849481446405 .loss/FBt3/detection_loss‚2.018677071801912 0loss/FBt4/assignment_loss‚0.3559489244915945ÿ¤(RÅÌ 0loss/FBt4/assignment_loss‚0.3559489244915945 .loss/FRt4/detection_loss‚2.547754468159242 $ _timestamp‚1.788441265673263e+09 /loss/FRt2/assignment_loss‚6.044729483755011 0loss/FBt3/assignment_loss‚0.7342745287266249 .loss/FRt1/detection_loss‚2.429312794390371 /loss/FRt3/detection_loss‚3.0026081284961186 /loss/SRqqt3/detection_loss‚2.79802034747216 /loss/FBt2/detection_loss‚2.0442936741992046 /loss/FBt4/detection_loss‚2.5070992298343455 epoch‚0 /loss/FRt1/assignment_loss‚5.684199502912619 1loss/SRqqt2/assignment_loss‚2.771911434084177 1loss/SRqqt1/detection_loss‚1.8776872882412539 /loss/FBt1/assignment_loss‚0.575081705847489 0loss/FBt2/assignment_loss‚0.3657507167982706 trainer/global_step‚1149 _step‚91 /loss/FRt3/assignment_loss‚6.262602239041715 /loss/FRt4/assignment_loss‚5.081446838378906 1loss/SRqqt4/assignment_loss‚2.809407435275696 0loss/SRqqt2/detection_loss‚4.496247085928917 2loss/SRqqt3/assignment_loss‚3.3675556336679766 _runtime‚ 982.720380663 2loss/SRqqt1/assignment_loss‚2.0076165827405124 .loss/FRt2/detection_loss‚2.571708792134335 1loss/SRqqt4/detection_loss‚1.4684118766869818 %loss/total_loss‚68.65963073121007 /loss/FBt1/detection_loss‚4.8372849481446405 .loss/FBt3/detection_loss‚2.018677071801912[ƒ®MDžÆ‚(:“ ¸ååÔЙýÇ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îQaÖLÇ‚(:Á ¸ååÔØé³É% disk.nvme0n1.out‚3362.73681640625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0* disk./.usagePercent‚19.470112584740544 disk.rbd1.out‚ 6.3076171875 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2629.84375 disk.drbd2080.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.051292419433594 cpu‚25.04627660884386 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.dm-1.out‚ 35.79296875 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 network.sent‚ 1.453276e+06 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚3362.73681640625 disk.sda.out‚ 35.79296875 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚411212% disk./.usageGB‚290.11874771118164 disk.rbd1.in‚0" disk.dm-0.out‚3362.73681640625 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125$ memory_percent‚95.13077735900879' proc.memory.availableMB‚ 797.7734375 disk.drbd2265.out‚0 disk.sda3.out‚ 35.79296875…¸F-’È‚(:‡ ÇååÔ¡¼§Ç gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.695' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.139 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë<¹õAÉ‚(:¶ ÇååÔü€—Ê disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.051292419433594 network.recv‚416249$ disk./.usageGB‚290.1285171508789 disk.sda3.out‚36.21875 disk.dm-1.out‚36.21875 disk.drbd2266.out‚0" disk.dm-0.out‚3417.78369140625 disk.drbd2265.out‚0 disk.rbd1.out‚ 6.3935546875' proc.memory.availableMB‚ 797.7734375 cpu‚25.04629196535667) disk./.usagePercent‚19.47076822003573 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚3417.78369140625 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.502867e+06 disk.drbd2080.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.13077735900879 proc.memory.rssMB‚ 2629.84375' disk.nvme0c0n1.out‚3417.78369140625 disk.sda.out‚36.21875 disk.sda2.out‚0þ ¶žÊ‚(:“ ÖååÔ˜¬•Ç gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.397' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.079400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ Ã‚BË‚(:· ÖååÔ€¡Ê network.recv‚421214 disk.dm-1.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚3483.693359375 disk.sda3.out‚ 36.7109375 cpu‚25.046341278641833 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 6.6513671875 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚95.13077735900879$ disk./.usageGB‚290.1427574157715 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda.in‚0) disk./.usagePercent‚19.47172389616105 disk.dm-0.out‚3483.693359375 disk.drbd2265.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.051292419433594 disk.dm-1.out‚ 36.7109375 disk.drbd2266.in‚0' proc.memory.availableMB‚ 797.7734375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 36.7109375 disk.sda3.in‚0 proc.memory.rssMB‚ 2629.84375 network.sent‚ 1.509337e+06# disk.nvme0n1.out‚3483.693359375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0]RË2Ì- _wandbruntime‚1024  _runtime‚1024w`ùÈÍ _step‚92 ' lr-AdamW/pg1‚0.00015276601550847065 ' lr-AdamW/pg2‚0.00015276601550847065 trainer/global_step‚1199 % _timestamp‚1.7884413072611935e+09 _runtime‚1024.308409234dÌ·ÌÎÆ _step‚92 ' lr-AdamW/pg1‚0.00015276601550847065 ' lr-AdamW/pg2‚0.00015276601550847065 trainer/global_step‚1199 % _timestamp‚1.7884413072611935e+09 _runtime‚1024.308409234\ä*Ë2Ï- _wandbruntime‚1024  _runtime‚1024«¦3©ÐÕO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81309593280050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82378855060992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83491823689216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82378837890560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82378837890560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186525696 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20457180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25695318 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90840215918592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1109511297586 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25695318 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1108730104370 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _step‚93 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1170962885632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81308795047474 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11999772 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46152500 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83492623218176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6114342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20457182 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83492623218176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1171149411328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚1024.309304674 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6114879 trainer/global_step‚1199%«*²ß'ÑÙO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25695318 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1108730104370 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _step‚93 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1170962885632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81308795047474 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11999772 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21614 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46152500 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26719 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21612 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83492623218176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26470 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6114342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 214063616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20457182 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26470 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83492623218176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1171149411328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781193216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚1024.309304674 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6114879 trainer/global_step‚1199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1020421120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81309593280050 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1020421120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82378855060992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 799528960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83491823689216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82378837890560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82378837890560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12012354048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186525696 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20457180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25695318 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90840215918592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1109511297586 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489]Ë Xu2Ò- _wandbruntime‚1025  _runtime‚1025ØÀ'¼ä&ÓÞM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20474220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1113913284096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46190900 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90915898108928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20474220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1114695655936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12009764 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6119510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82447469464064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25716678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46169260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6120040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚94 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1113913284096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 $ _timestamp‚1.788441308068616e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83561382748160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83562182290432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20474222 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82447486634496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18129274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1171947425280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81377335025590 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1110435254198 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81376536792806 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82447486634496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90915922881536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1114695655936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25695040pR϶ÔâM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83561382748160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83562182290432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20474222 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82447486634496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18129274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1171947425280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81377335025590 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1110435254198 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81376536792806 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82447486634496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90915922881536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1114695655936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25695040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20474220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 799542272 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21638 XFDeviceStatsMonitor.ožÏñY2n_train_batch_end/allocated_bytes.small_pool.freed‚ 1113913284096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46190900 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26470 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213382144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90915898108928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781193424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20474220 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1114695655936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12009764 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6119510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12012354048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 782371840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 214063616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82447469464064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25716678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46169260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1020421120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6120040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188609536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚94 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1113913284096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798232784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635023317 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019218389 $ _timestamp‚1.788441308068616e+09^0È2o2Õ- _wandbruntime‚1025  _runtime‚1025 í¦LTÖÎ /loss/FRt4/assignment_loss‚4.787434781751325 1loss/SRqqt1/detection_loss‚2.3838848175687923 0loss/SRqqt3/detection_loss‚3.286824357509613 epoch‚0 1loss/SRqqt3/assignment_loss‚3.011152458190918 1loss/SRqqt4/detection_loss‚1.2722001660908206 /loss/FBt2/detection_loss‚2.2434234324394278 _step‚95 0loss/FRt2/assignment_loss‚6.1249004772731235 2loss/SRqqt1/assignment_loss‚1.9970107911913484 0loss/FBt3/assignment_loss‚0.6183232523214358 trainer/global_step‚1199 /loss/FRt1/assignment_loss‚5.847460185156928 2loss/SRqqt2/assignment_loss‚2.5856252987133828 .loss/FRt1/detection_loss‚2.978403489788373 /loss/FRt3/detection_loss‚3.3597244968016944 /loss/FRt3/assignment_loss‚6.550659434000651 0loss/FBt1/assignment_loss‚0.8711081397746057 /loss/FRt4/detection_loss‚2.1869489108362505 % _timestamp‚1.7884413080701628e+09 0loss/SRqqt2/detection_loss‚4.314863982953523 .loss/FBt4/detection_loss‚1.397543625617534 0loss/FBt2/assignment_loss‚0.3766830039719143 0loss/FBt4/assignment_loss‚0.6021982885657304 .loss/FBt3/detection_loss‚4.878134596251095 1loss/SRqqt4/assignment_loss‚2.530639923020695 /loss/FRt2/detection_loss‚2.8295135838644847 /loss/FBt1/detection_loss‚6.4828473925590515 %loss/total_loss‚73.51750888621272 _runtime‚1025.117258826a X×Ò 1loss/SRqqt4/detection_loss‚1.2722001660908206 /loss/FBt2/detection_loss‚2.2434234324394278 _step‚95 0loss/FRt2/assignment_loss‚6.1249004772731235 2loss/SRqqt1/assignment_loss‚1.9970107911913484 0loss/FBt3/assignment_loss‚0.6183232523214358 trainer/global_step‚1199 /loss/FRt1/assignment_loss‚5.847460185156928 2loss/SRqqt2/assignment_loss‚2.5856252987133828 .loss/FRt1/detection_loss‚2.978403489788373 /loss/FRt3/detection_loss‚3.3597244968016944 /loss/FRt3/assignment_loss‚6.550659434000651 0loss/FBt1/assignment_loss‚0.8711081397746057 /loss/FRt4/detection_loss‚2.1869489108362505 % _timestamp‚1.7884413080701628e+09 0loss/SRqqt2/detection_loss‚4.314863982953523 .loss/FBt4/detection_loss‚1.397543625617534 0loss/FBt2/assignment_loss‚0.3766830039719143 0loss/FBt4/assignment_loss‚0.6021982885657304 .loss/FBt3/detection_loss‚4.878134596251095 1loss/SRqqt4/assignment_loss‚2.530639923020695 /loss/FRt2/detection_loss‚2.8295135838644847 /loss/FBt1/detection_loss‚6.4828473925590515 %loss/total_loss‚73.51750888621272 _runtime‚1025.117258826 /loss/FRt4/assignment_loss‚4.787434781751325 1loss/SRqqt1/detection_loss‚2.3838848175687923 0loss/SRqqt3/detection_loss‚3.286824357509613 epoch‚0 1loss/SRqqt3/assignment_loss‚3.011152458190918_- Í’Ø‚(:‡ åååÔù´´Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.653' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÙ>s0Ù‚(:¥ åååÔ·Þ«É disk.dm-1.in‚0 network.recv‚424792 disk.drbd2080.in‚0 disk.rbd1.out‚ 6.7373046875 disk.sda1.in‚0 disk.sda.in‚0$ proc.memory.availableMB‚722.4375# proc.memory.rssMB‚ 2704.83203125 disk.dm-1.out‚37.5$ memory_percent‚95.59059143066406* proc.memory.percent‚16.508984565734863 cpu‚25.05609750050833% disk.nvme0c0n1.out‚3540.349609375 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda.out‚37.5 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.nvme0c0n1.in‚0.078125 disk.rbd0.in‚0 disk.sda2.in‚0* disk./.usagePercent‚19.472479630402127$ disk./.usageGB‚290.1540184020996 disk.rbd0.out‚0# disk.nvme0n1.out‚3540.349609375 disk.dm-0.in‚0.078125 network.sent‚ 1.514601e+06 disk.drbd2080.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚37.5 disk.dm-0.out‚3540.349609375ReD‘Ú‚(:† ôååÔÀó±Ç gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚198.66' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏįOMÛ‚(: ôååÔ•™É disk.dm-0.in‚0.078125 disk.sda3.out‚ 37.9921875 disk.dm-1.in‚0" disk.dm-0.out‚3607.38330078125 disk.rbd1.out‚ 6.7998046875 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0$ memory_percent‚95.59276103973389" proc.memory.rssMB‚ 2705.1171875 proc.cpu.threads‚17 disk.sda.out‚ 37.9921875 disk.sda3.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚3607.38330078125 disk.sda1.in‚0 cpu‚25.121302896800245 network.sent‚ 1.564474e+06 network.recv‚430631 disk.drbd1716.in‚0* proc.memory.percent‚16.510725021362305* disk./.usagePercent‚19.473810869580177 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0( proc.memory.availableMB‚ 722.08203125% disk./.usageGB‚290.17385482788086 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚3607.38330078125 disk.dm-1.out‚ 37.9921875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0,E’Ü‚(:‡ ƒæåÔ×ÁÉÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.397' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øA­GÝ‚(:¼ ƒæåÔºÄÅÉ proc.cpu.threads‚17 network.sent‚ 1.570268e+06 disk.dm-0.in‚0.078125 disk.sda.in‚0 disk.sda.out‚ 38.703125 disk.sda2.out‚0$ memory_percent‚95.59571743011475( proc.memory.availableMB‚ 721.59765625) proc.memory.percent‚16.51332378387451 disk.drbd1716.in‚0% disk.nvme0n1.out‚3664.06298828125 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2705.54296875 cpu‚25.18492077668789 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚290.1463928222656 disk.sda3.out‚ 38.703125 disk.rbd1.out‚ 6.7998046875 disk.dm-1.out‚ 38.703125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3664.06298828125 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125) disk./.usagePercent‚19.47196787134118" disk.dm-0.out‚3664.06298828125 disk.sda1.in‚0 network.recv‚434991 disk.dm-1.in‚0 disk.sda2.in‚0Àð–’Þ‚(:‡ ’æåÔçîÑÇ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚36.0 gpu.0.powerWatts‚188.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.7944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ ÍëEß‚(:º ’æåÔìø°É disk.sda.in‚0 disk.dm-1.out‚ 39.17578125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 39.17578125 disk.nvme0n1.in‚0.078125# disk.nvme0n1.out‚3726.294921875 disk.drbd2080.in‚0 network.recv‚439620) disk./.usagePercent‚19.47613977011916 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 6.94140625( proc.memory.availableMB‚ 721.52734375 proc.memory.rssMB‚ 2705.59375% disk.nvme0c0n1.out‚3726.294921875 disk.sda.out‚ 39.17578125 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125$ memory_percent‚95.59614658355713 cpu‚25.245029345047932 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0% disk./.usageGB‚290.20855712890625 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.513633728027344 disk.dm-0.out‚3726.294921875 disk.drbd2266.in‚0 network.sent‚ 1.576526e+06 disk.sda1.in‚0 disk.sda1.out‚0ö»|nàbi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP~gùSxábs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPX&¡oâbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPÅíÉ~yãbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPœÐöšyäbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPRßÙÃwåbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP«yS2uæbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPþ4~çby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPâg†qèbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JP â&Ötébo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPŠˆª têbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP$æ·%wëbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPFbÆlìbg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP^5Ytíbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP¾º»…wîbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPü¨¿\xïbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPšœÀfðba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPÜXQ^nñbi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JPI †\fòba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPÈf±°dób_ œ…UcÑb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPõݤÒrÒbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP+€uÓbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP%TîtÔbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP•þ »lÕbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP¶Âp„kÖbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPï9ü¶v×bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÇþ€ZxØbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP@ ôévÙbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPG³#ü~Úby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPó•Œ42Û- _wandbruntime‚1084  _runtime‚1084¶K› ܼT FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0¯û€F*ÝÀT aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚85275941412352 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚86418057495552 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚26272460 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚86418057495552 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 604670464 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3157 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 603022732 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚17495 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚18900180 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚93855993253888 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11635023317 trainer/global_step‚1208 % _timestamp‚1.7884413670805452e+09 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 _runtime‚1084.128222112 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1010 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚82963892323328 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12259950592 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚6391468 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1010 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚85275941412352 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚6391289 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12259950592 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1010 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1020421120 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1141528583168 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚17495 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚95056195220992 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 Frõbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÚ#ˆnöbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPΑ(Ps÷bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPⶹsøbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP ã«;jùbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP!M¸[úbV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP8œÿvûbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP{ö)¤qübl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPTŒîjýbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP}µÂrrþbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPB×ørÿbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP³k>Øu€bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPqyé³jbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP¶0{¢r‚bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP j½wƒbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÿ×ô6c„b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPšÉù‡o…bj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÿ*ìÌj†be BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPÌaŠºr‡bm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPqÑ ºeˆb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPÊ¡| k‰bf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPÆ mâuŠbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPqÜ”n‹bi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JPšaW mŒbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPç«Ýeibd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPrsà5kŽbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPs6öÎpbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP«MErbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¼²Hlm‘bh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPi~Gt’bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPÒ³àÑ|“bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPÁªyr”bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPƒ)è¬m•bh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPxm­n–bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPüC2Vn—bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPZ{3l˜bg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP “?h™bc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPÛÉ™Žnšbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP­Èîk›bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP¥š%qœbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPÄÓ-kbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP­ÄÆ7džb_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPwæô|t bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP ŽÕ2s¡bn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP ŒDz¢bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPg•ÃÚb£b] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPLZi¤bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPB.‰w¥br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPõº´€m¦bh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JP¼³e¯i§bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPó¨m´p¨bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPÓg |©bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP:µÃgªbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP¹qr«bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPÑkcq¬bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPåÈ0ýz­bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP)((Œx®bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPìdg>w¯br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP;ó]Æl°bg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPQ¦DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPõôv¼bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP-U³¡u½bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÇy2r¾bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP™]j¿be BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPž7qOqÀbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP›6rÁbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP¬ ·æuÂbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP ®¬yrÃbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÃù½pÄbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP¯oÅbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPNrUkÆbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPB/òOjÇbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPþ_©rÈbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP×(U=cÉb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP±¨ÜwÊbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPñÑÍkgËbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JP"ÑqlÌbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPªES/cÍb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP©Ì§œsÎbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPŠí’/iÏbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPsÜ?jpÐbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP"&xaÑb\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPÕ# wÒbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP½àiÓbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPàèûßmÔbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JP0û ktÕbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPfš¸qÖbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP/ë—Ñk×bf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPáE™jØbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JP˜*œÞ2Ù- _wandbruntime‚1085  _runtime‚1085,øäyJÚÆR SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allH™ýî)ocated‚26297737 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚85321042009600 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1141917523968 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 _runtime‚1085.123692857 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚32 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚21283080 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚6396090 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚26280242 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1010 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚521 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚26297737 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚17497 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚47580819 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚95102685293056 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚183 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚26470 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 604670464 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚86463564204032 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚21283082 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚26470 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚180 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚93902119157760 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚12517242 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 603022732 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚21283082 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 12012354048 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 604670464 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11635023317 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26719 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 162887168 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 138114560 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚83006660332544 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚521 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚17495 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1200566135296 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚17170432 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚95102522405888 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚24772608 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚93902094385152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚84144192944474 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚86463564204032 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚26719 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚85321059180032 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1200428020736 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884413680759761e+09 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚85321059180032 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 585983372 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚521 trainer/global_step‚1208 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587500032 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 587500032 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12259950592 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚47563322 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1142505024000 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚47563322 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1010 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1137549651290 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚86462959533568 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1141917523968 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1138135634662 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚17497 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 _step‚97 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚18913332 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1051 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚85321042009600 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚18913515 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 12012354048 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371895808 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚84144795967206 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1020421120 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚21283080 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚17170432 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3157 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚12517245 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚26280242 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1010 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚47580819 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1142505024000 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚86462959533568 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26719 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 162887168 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 138114560 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚83006660332544 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚521 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚17495 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1200566135296 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚17170432 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚95102522405888 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚24772608 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚93902094385152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚84144192944474 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚86463564204032 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚26719 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚85321059180032 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1200428020736 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884413680759761e+09 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚85321059180032 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 585983372 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019218389 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚521 trainer/global_step‚1208 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587500032 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 587500032 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1020421120 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12259950592a[ãþžÜ‚(:“ ¡æåÔúìÑÇ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&HkzFÝ‚(:» ¡æåÔ¯ŠÓÉ disk.sda.out‚ 39.66015625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.078125 disk.rbd0.in‚0 network.sent‚ 1.581898e+06 disk.sda3.out‚ 39.66015625 disk.drbd1716.in‚0 disk.dm-0.out‚3774.638671875 disk.rbd1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.51463508605957# disk.nvme0n1.out‚3774.513671875 disk.drbd2265.out‚0 disk.rbd1.out‚ 6.94140625 disk.sda.in‚0 cpu‚25.305516141866214 network.recv‚443155* disk./.usagePercent‚19.474395846795918$ disk./.usageGB‚290.1825714111328 disk.dm-1.in‚0 disk.dm-1.out‚ 39.66015625 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2705.7578125 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚3774.462890625$ memory_percent‚95.59433460235596( proc.memory.availableMB‚ 721.82421875 disk.drbd1716.out‚0 disk.rbd0.out‚0è°³HÞbC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPé'ogHßbC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP}Å^í@àb; particle/accuracy_6_of_7"trainer/global_step2:JP™&Y&HábC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP7sYxJâbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP ©{®NãbI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¬ ½·NäbI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÙÔ•HåbC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP‰kK;æb6 jet/accuracy_3_of_6"trainer/global_step2:JPzÀ%@çb; particle/accuracy_4_of_7"trainer/global_step2:JP ËÚJèbE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPk¾NébI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÁ‰JêbE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPRå_<ëb7 particle/specificity"trainer/global_step2:JP‚À DHìbC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÏòvTHíbC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPíÙaJîbE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPt.]„JïbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPuH]JðbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPðECJñbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP&É2NòbI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÍëÏ;ób6 jet/accuracy_3_of_4"trainer/global_step2:JP4H!HôbC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPÕMíJõbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP­"®KHöbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPð0•N÷bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPýíóNøbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP<ý ”JùbE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPÌ4EJúbE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPëÙ‚JûbE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP­ý¦¬HübC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP±ÁpšNýbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP·N´­NþbI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPSLå¡JÿbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¦oÝN€bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÐf&ò;b6 jet/accuracy_1_of_5"trainer/global_step2:JPèû—š@‚b; particle/accuracy_1_of_1"trainer/global_step2:JPèÌJƒbE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP@nH„bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP!j¿?@…b; particle/accuracy_6_of_6"trainer/global_step2:JPÁØ6;H†bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP6þQªJ‡bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÔÚÁJˆbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÄñœTJ‰bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP´«NŠbI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPpRö<@‹b; particle/accuracy_5_of_6"trainer/global_step2:JP«Ì?HŒbC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP§t¤½HbC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP±zÁHŽbC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¯àqêNbI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÔëÍÏHbC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP]‰ÓxH‘bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPʼûNN’bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP7._WJ“bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPwݵJ”bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP%u[H•bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPµí†³H–bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP1q°ýJ—bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPì£y@H˜bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPM2ëH™bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP‘mdmJšbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPpzJ›bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÙÙbNœbI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPrì¾NbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÉ÷#>;žb6 jet/accuracy_4_of_5"trainer/global_step2:JPÅY JŸbE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPrº–H bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPs™×BH¡bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPdÛ€N¢bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÍwºPJ£bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPЇ*ÉH¤bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPè^€J¥bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPš†$}J¦bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPs…N§bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPõ:IN¨bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPŽWVN©bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP©1CÄHªbC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÄ1ªSH«bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPà{ÝH¬bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPH2ÀäN­bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP´÷T'J®bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP„wî–N¯bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPËM`UN°bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPV@'QN±bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP8Ï"ƒH²bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP7<¸N³bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPüXëJ´bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPO¯•£NµbI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP‚»åJ¶bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPzqänH·bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP<¢,ÏJ¸bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP^›¬²N¹bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÄ’ÜÓJºbE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPMÒ©µN»bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPZfÓÔN¼bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP±vÞiH½bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPHâOÁ;¾b6 jet/accuracy_2_of_6"trainer/global_step2:JP¥(HfH¿bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP«.Ÿ³NÀbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¯lº;JÁbE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPž›ó£HÂbC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPŸ¸,wHÃbC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPHªŽ÷HÄbC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPx´$?HÅbC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPbÿÔNÆbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÊ):NÇbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPcCôšNÈbI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPs,ñÙJÉbE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP3BÄ®JÊbE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPléÓÒHËbC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP-[ùHÌbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP°APgJÍbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP·¤ ×JÎbE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP„ÿM"JÏbE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPUlV”HÐbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP»Šn™NÑbI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP8@WNÒbI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPƒÐk•8Ób3 particle/f_score"trainer/global_step2:JP‚ôhÆJÔbE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP*k'.NÕbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¯Ù¢HÖbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP±ªÖŽN×bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPmƒžÅHØbC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPë¨aHÙbC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP{çuÚHÚbC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPWB–NÛbI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPJ/|(ÜbI &Purity/4FRt0SRqqt1FBt/event_propoP w#&rtion"trainer/global_step2:JPÝ*é²JÝbE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPM¨ã JÞbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPMÛT HßbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPËLÔÌNàbI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPðËÓ‡NábI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPúÕ]JâbE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¥¶JãbE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPHá|ÞJäbE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPCô^NåbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÏ4—¨;æb6 jet/accuracy_3_of_5"trainer/global_step2:JP†üHçbC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP¿‘éÓJèbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP¾cqJébE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸…‹YNêbI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPAHæJëbE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPV2ˆšJìbE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP'±r4NíbI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPj9é/@îb; particle/accuracy_1_of_6"trainer/global_step2:JP=¯çxJïbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP%‡KdJðbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƒPCJñbE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPº‡üNòbI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPàÇ”UHóbC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPèw¿èNôbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPLü´NõbI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPvì!NöbI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÒ¢YcJ÷bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPí¶JøbE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP3oHùbC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP9c}_JúbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP }ºÝHûbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPùÄ NübI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP?ÚNýbI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPm5}SJþbE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP«IÆXJÿbE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPl’UJ€ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP†æ$¶J bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPpŠ]J‚ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP…»OJƒ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ë#N„ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP/ÚA¤J… bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPõQJFJ† bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÅZ®§N‡ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPd‹ñNˆ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP†Å”H‰ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPz¥5¶NŠ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP! N‹ bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP.?X?HŒ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP5­TJ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPhñJŽ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPâf†Ö@ b; particle/accuracy_1_of_7"trainer/global_step2:JPíu†H bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP—qÒÄH‘ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP‹ž ¹J’ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP„áåJ“ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‚è‚H” bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÝç¹7J• bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP†y³H– bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPØ(ÏJ— bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ½~{H˜ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP]ÂÓN™ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPùÜ ‘Jš bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP­7ÝWH› bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP…˜.Jœ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPgðT¢H bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP˱êØHž bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPú] ½NŸ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPCÁr·N  bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPb哹N¡ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP*WĤJ¢ bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP&Œ¤ÙN£ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¦ŒîBN¤ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPNw¼kJ¥ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPðþ ýH¦ bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPmþäN§ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP^ÏN¨ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPи’N© bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPûFÚ;ª b6 jet/accuracy_2_of_4"trainer/global_step2:JP¡’ùÈN« bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPf°‡@¬ b; particle/accuracy_1_of_2"trainer/global_step2:JPc¶”õG­ bB validation_average_jet_accuracy"trainer/global_step2:JPçëî²H® bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP„ár?J¯ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÈŸJ° bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPnÞÃH± bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP }™;² b6 jet/accuracy_7_of_7"trainer/global_step2:JPš£=dH³ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP,Äà'N´ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPŠ˜ríJµ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPWŸÏ‘H¶ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP¾ï{‚J· bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‚ פH¸ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP \œ+J¹ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPæиÃHº bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPDˆe‚J» bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPu'brJ¼ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPµÔb­H½ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP‹¬˜•J¾ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP2§N¿ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP å‘JÀ bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPEöÑ=NÁ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP“ÜGìH bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP’ÿ˜8Hà bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPl'D†JÄ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP}HjJÅ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¶—ü?@Æ b; particle/accuracy_7_of_7"trainer/global_step2:JP>(”ªNÇ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP²¤3HÈ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPGÌÉNÉ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÌÈ àJÊ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÚÈØQJË bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPƒÊàmJÌ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÞÅpJÍ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPe]æáHÎ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP°!¼JÏ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP÷7¹JÐ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP³²HÑ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¬K6»;Ò b6 jet/accuracy_1_of_2"trainer/global_step2:JPÕdÃU@Ó b; particle/accuracy_3_of_5"trainer/global_step2:JP&$ò9Ô b4 particle/accuracy"trainer/global_step2:JP¸£ÏHÕ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP ~tHÖ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP4YÏþH× bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP&a—@Ø b; particle/accuracy_3_of_3"trainer/global_step2:JPÓ‘RæJÙ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP/±,#NÚ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPuë3¦NÛ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÑ綗NÜ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP~—C¶HÝ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP{Æ;Þ b6 jet/accuracy_1_of_7"trainer/global_step2:JPÄR¬Jß bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP¯q¿Nà bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP’góNá bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÄUdHâ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPKÿôNã bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP} :Hä bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPu5‘Nå bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPˆñ>ANæ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP6;›XNç bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPêíÁ>Hè bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPâFiÆHé bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÿ€5Hê bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP*œ´6Jë bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP"‡7ñNì bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP…z¼»@í b; particle/accuracy_2_of_6"trainer/global_step2:JP“>ßJî bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPc@!×Nï bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP'“pHð bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPHæ~Nñ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP™› Nò bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPšŽ#¶Nó bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP`#”Jô bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPmòê£Nõ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP1ÃO4Hö bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPžû?N÷ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÛ1ØHø bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP#p¢¯Jù bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÂߤ½@ú b; particle/accuracy_1_of_3"trainer/global_step2:JPòó÷Hû bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP@/uHü bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPëz:Jý bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPŒòW(Hþ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPæð‘‡Nÿ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÎÈåÃJ€ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPƬ6H bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPçÃ…´N‚ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP_qœHƒ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP~R*J„ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP¦lïÆN… bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP;dePN† bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÂ`o1J‡ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPšíHˆ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPòy MH‰ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPbáÔöHŠ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPfxxèJ‹ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPä)ÈNŒ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP]Æ8ÛH bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPª(xÓJŽ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP(×ß7J bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP–\ÒÚJ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP­Ú]J‘ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÆ~ŸÂH’ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP˜N“ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP‚[ܘJ” bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP›"`J• bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP–˜¯J– bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP„îÿÝN— bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÏÿAJ˜ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP©áPoN™ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP“ųNš bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÉÇÚ6N› bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPË„b9Nœ bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPEUŒ3J bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPò ÓäJž bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP«}·HŸ bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÜÌæÛ@  b; particle/accuracy_2_of_5"trainer/global_step2:JP=v®J¡ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPd¤ð J¢ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPþ{ƯJ£ bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP…ÝÀ¯J¤ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP}ljN¥ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPµ4uïN¦ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP>áÈÏ@§ b; particle/accuracy_4_of_5"trainer/global_step2:JPJгYJ¨ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP‚yõ@H© bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÌRlúJª bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPjÂ<ÉN« bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¨„ºÉJ¬ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP™#\J­ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP§ÓÍTH® bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP&qò¹J¯ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP©2BÜN° bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPi¾ŸN± bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPB—N² bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP +_ú@³ b; particle/accuracy_2_of_3"trainer/global_step2:JPÔÐJ´ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÿb]—Jµ bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP”18`J¶ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPôÀ¥"H· bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPy@J`@¸ b; particle/accuracy_4_of_4"trainer/global_step2:JPQy£;¹ b6 jet/accuracy_4_of_6"trainer/global_step2:JP*ŒwNº bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP@âJ» bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPD†·N¼ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP]¬ N½ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÀ˜Ù:H¾ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPŠjFJ¿ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPº»¤HÀ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPSFwHÁ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPB\AYN bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÀ} Jà bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPó÷•œNÄ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPGr7îHÅ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP“’mVNÆ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÎ)ʘNÇ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¾$b3JÈ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPʹbDHÉ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP6 NÊ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP±¬ÞwJË bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPƒ+Ó»JÌ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÖE2HÍ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPv,×¾JÎ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP-ûÕNÏ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPm•þNÐ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÔ)Ñ´JÑ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP:ó”‹HÒ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPöˆÜ‚JÓ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP±ý¦ÜJÔ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP}ÙiJÕ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP»’ƒ@Ö b; particle/accuracy_1_of_5"trainer/global_step2:JPUÆH× bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPˆÏVHØ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPpé¥éNÙ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP"|eNÚ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¶v%HÛ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP€MÄH;Ü b6 jet/accuracy_6_of_7"trainer/global_step2:JP·‹æNÝ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP\ÿZ?HÞ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP9…¯tJß bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP½ËÖ$Hà bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPŸ5Ná bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÂÇ}Nâ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPñ»\›Jã bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPŸÆ™Jä bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPcÒÇNå bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¥'™Næ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPEhË×Hç bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃRÉHè bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÂqÂHé bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPVU¬Jê bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¢ÀXJë bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÚ²ÔHì bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPá}GEJí bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÿ€¢0Jî bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÿÀJï bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPJ wÇJð bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP )2‹Nñ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPê­Nò bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPZ |Jó bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP’²Ó¨Hô bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒ$oNõ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP!qDHö bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPìJ¾J÷ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPUT)Jø bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP¬”ç•Hù bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPd'o]Jú bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÚû²VNû bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPiÿžNü bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¹vóžNý bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPù2ÙJþ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPóVã;ÿ b6 jet/accuracy_1_of_1"trainer/global_step2:JP‡9ðJ€ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP ]KËH bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP˜ºR5J‚ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP|×K,Jƒ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP÷^°‡N„ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPâÁgH… bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP‘zÜH† bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP«H/H‡ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÐܦNˆ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÅ0òËN‰ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPûˆÉJŠ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP(þú•@‹ b; particle/accuracy_1_of_4"trainer/global_step2:JP:|JŒ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÖ` ôJ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÖ(ÿmNŽ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŸ½N bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP,e N bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÈg:J‘ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP£ÃÍJ’ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP`u›gH“ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP»]ÿÍH” bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP¯¼H• bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP+xÕJ– bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPyXKÝH— bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP_ ŠzN˜ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP± üq;™ b6 jet/accuracy_4_of_7"trainer/global_step2:JPø9?äJš bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP’h‡H› bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP€ÑÌûJœ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP²d5÷H bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPJóü Hž bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPKÐ#ôHŸ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPW5¶ËJ  bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPë7ÈèN¡ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPBqø^N¢ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÿ#1J£ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPïôãÞH¤ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÇÛN¥ bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP;`ég@¦ b; particle/accuracy_5_of_5"trainer/global_step2:JPNª&vJ§ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPC™,J¨ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP–³ËÒ@© b; particle/accuracy_3_of_7"trainer/global_step2:JPq6âTNª bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPèöBSN« bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPó½ðÍJ¬ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÌxÅH­ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP¡M¹H® bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒÖeN¯ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÿüÕJ° bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPûw},H± bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¹ªH² bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¸ uDH³ bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÃLJ´ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP2ØêNµ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP>Ýå/J¶ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP'! H· bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP®˜\êH¸ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPlÌdJ¹ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPjcJº bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPé_ì’J» bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‰ oûN¼ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP…Ä¥AJ½ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP["XçH¾ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP@êVH¿ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP€¥ÏNÀ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPI¸»[HÁ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¤—>H bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP¢º¡Hà bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPý“δ@Ä b; particle/accuracy_5_of_7"trainer/global_step2:JP±BÙÎJÅ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP ²¸“HÆ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP·­·HÇ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPð$UDNÈ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPý…‡JÉ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœ4ºNÊ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP‹¹HË bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP°†tJÌ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPû!d JÍ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPýŒ²wNÎ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPh±lJÏ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPDBÀNÐ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP;{ŠvJÑ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPŠ Ž…JÒ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPF ÷ÁJÓ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPÛæWÔNÔ bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPˆ–ðNÕ bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP²ÙŒš@Ö b; particle/accuracy_2_of_2"trainer/global_step2:JP>èŽH× bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPw†¤@Ø b; particle/accuracy_4_of_6"trainer/global_step2:JP´_9;Ù b6 jet/accuracy_5_of_7"trainer/global_step2:JP3Ó¼WJÚ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPR¢„HÛ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPͼ£áJÜ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPˆ¥gÃHÝ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÄsINÞ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP[”ÏNß bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPØ ¡“Jà bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸ¤3Há bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP]Ð \Nâ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPN¡zJã bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÑvÐwNä bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP3„!Nå bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPïp®Næ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP u^ Hç bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP½rý Hè bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP`ØËJé bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPìèNê bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP³¬äyJë bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPùBJì bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP—»mÛJí bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³È*"Hî bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÏ}”I;ï b6 jet/accuracy_1_of_4"trainer/global_step2:JPèY‡Q;ð b6 jet/accuracy_6_of_6"trainer/global_step2:JPOêË÷Hñ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP2&š7ò bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/globalÛ­4E_step2:JP¿¼}Jó bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPuÓ‚wNô bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÝÈÛNõ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÙRtèNö bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP×É ˜N÷ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPgHø bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÚ…Ÿ=Jù bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPf¾ðNú bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPW×IlJû bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPn‰ËÛNü bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPST£]Jý bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPœÒNþ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPQ=:èJÿ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP…S0tH€ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPQÿwJ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPñJ iN‚ bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPûuuuNƒ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP;MËb<„ b7 particle/sensitivity"trainer/global_step2:JP lxJ… bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP<•èH† bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP Ø» N‡ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPR@ÝeJˆ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPãi >@‰ b; particle/accuracy_2_of_7"trainer/global_step2:JPÃʱHŠ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP‡;7àH‹ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¹ÈêNŒ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPn4v¸J bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÎaÀ»HŽ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPqHzŒN bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPZk=J bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPs ì#H‘ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔm ¢;’ b6 jet/accuracy_2_of_7"trainer/global_step2:JPAøÛx;“ b6 loss/val_total_loss"trainer/global_step2:JPìÓõ›N” bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP(Ä…J• bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP}>.N– bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPN{6H— bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP(t2òH˜ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP´mÒüJ™ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPiRlHš bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPa;ŒåN› bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPd³Ù Jœ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀ®á’H bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP¡ø,:Jž bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPß=æJŸ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPk•!—H  bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPŠ®ÆH¡ bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPën=J¢ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPxŒ<2J£ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP«ušN¤ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÚ”tH¥ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP¹·?J¦ bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP3çN§ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP®ŽŽ¦J¨ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPc²m€J© bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¦/IrJª bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP[BûìH« bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPO®N¬ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP®çtN­ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP—Þ"N® bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¥©2OH¯ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPî¼MJ° bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP"{‡ßN± bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÔ¤;² b6 jet/accuracy_2_of_5"trainer/global_step2:JPÞ+;³ b6 jet/accuracy_5_of_6"trainer/global_step2:JP»jŽh@´ b; particle/accuracy_3_of_6"trainer/global_step2:JPé]H­Nµ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPdÒÊÀN¶ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPö;Š;J· bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP-½½óN¸ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPë–’ìJ¹ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPå+ÃHº bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPa•N» bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPåü(ñN¼ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÚœ]EN½ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPO¢8ÞJ¾ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸æ¯N¿ bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¤å¹ÿNÀ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPd¯P±NÁ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPAøƒßH bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP® [`Jà bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP?kÎNÄ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPóNÅ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP<ÛÅHÆ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPˆÎx;Ç b6 jet/accuracy_4_of_4"trainer/global_step2:JP´œ*fJÈ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPq]xkJÉ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP%9Í1JÊ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPExHË bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¢ˆÉHÌ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPWHÍ bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPŵ=NÎ bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPk5ÝõHÏ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÖâ‰9HÐ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP™F6JÑ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP•þ¸xN§ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPQ}} J¨ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP4ݾïJ© bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPa.õŽNª bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPM¦¥N« bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPNŒH¬ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP0ôð2­ - _wandbruntime‚1098  _runtime‚1098­MònE® Ä£ =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FRt_purity‚0 9 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.016203705221414566 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.032393928617239 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0011691674590110779 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.10493826866149902 +jet/accuracy_2_of_2‚0.04096266254782677 jet/accuracy_5_of_5‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9737046957015991 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0004409171233419329 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.11206896603107452 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 particle/accuracy_6_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 ,jet/accuracy_3_of_6‚0.012820512987673283 particle/accuracy_4_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8904691934585571 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.03860034421086311 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.10876043885946274 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.02859094738960266 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.01383029855787754 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.00290291546843946 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9537158608436584 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.7317140102386475 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9366326332092285 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0011574075324460864 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.3106060326099396 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.14887550473213196 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1101158931851387 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 : Purity/3FRt*SRqqt1FBt/FRt_purity‚0.0055555556900799274 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.09886793047189713 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3768160343170166 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚1098.425279119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9772205948829651 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.059950679540634155 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.26710575819015503 9 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.037748806178569794 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/3FRt3SRqqt*FBt/FRt_purity‚0 & Purity/1FRt*SRqqt3FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.02557678520679474 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1272840052843094 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2916208505630493 9 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.022972973063588142 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.11481481045484543 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.1666666567325592 9 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.030265051871538162 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.17901232838630676 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.0625 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.08812636137008667 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9941788911819458 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2271825224161148 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.03231654316186905 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.25738784670829773 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.03076357953250408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.040672462433576584 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.014880952425301075 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.12726861238479614 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9534366726875305 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.09723792970180511 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 +jet/accuracy_3_of_5‚0.01666666567325592 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.30701756477355957 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.11666666716337204 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.10815619677305222 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.07394636422395706 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7884413813758066e+09 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07121039927005768 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.26403144001960754 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.1465441882610321 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10335942357778549 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.002543752547353506 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05246913805603981 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.103050097823143 particle/accuracy_1_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.164198100566864 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.31874996423721313 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.962355375289917 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.15848568081855774 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.044312167912721634 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.30147767066955566 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03273829445242882 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.1274973452091217 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.25754597783088684 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9049707651138306 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9906757473945618 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.025531597435474396 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.08172816783189774 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 trainer/global_step‚1208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9774238467216492 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.05211000144481659 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33624330163002014 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3218509256839752 particle/accuracy_7_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/*FRt0SRqqt3FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.14561916887760162 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.013792130164802074 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.05614035576581955 +jet/accuracy_1_of_2‚0.48445719480514526 particle/accuracy_3_of_5‚1 (particle/accuracy‚0.7881684899330139 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9742642045021057 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.04213183373212814 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.038018740713596344 particle/accuracy_3_of_3‚1 _step‚98 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.011005891487002373 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 jet/accuracy_1_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.04798850789666176 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.016071714460849762 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9287253618240356 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9913687109947205 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.013291110284626484 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.992583155632019 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03187708184123039 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9741886854171753 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.05652341619133949 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.06493055820465088 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.053614940494298935 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.06042885780334473 9 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.019086651504039764 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2652789354324341 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.000372023816453292'u-Y­L97 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.18103446066379547 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846491813659668 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_4_of_5‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011066308245062828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9753362536430359 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.21759259700775146 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.5998294353485107 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9949758648872375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3496510684490204 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.036654695868492126 particle/accuracy_4_of_4‚1 jet/accuracy_4_of_6‚0 epoch‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FRt_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.1908131241798401 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.04545454680919647 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05268199369311333 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 +"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.11689426749944687 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9865273237228394 <"Purity/1FRt*SRqqt0FBt/event_purity‚0.0058403233997523785 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2645833492279053 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9561861753463745 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.07655038684606552 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9624934792518616 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0038548754528164864 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FRt_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2883397936820984 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15935011208057404 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.11689426749944687 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11303525418043137 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.039019130170345306 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.276717871427536 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.03533031791448593 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12284865975379944 +jet/accuracy_1_of_1‚0.27732330560684204 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9876915216445923 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.05321640148758888 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013785468181595206 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.11202432215213776 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1770833283662796 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.024154843762516975 & Purity/1FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.03556547686457634 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.013291110284626484 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.25816962122917175 & Purity/*FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887377619743347 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.02978750877082348 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.025757577270269394 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13772046566009521 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.033551864326000214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9898214340209961 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.08251488208770752 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2222016304731369 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24947331845760345 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9592934846878052 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.032568030059337616 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt3SRqqt0FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9953703880310059 particle/accuracy_5_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.034246791154146194 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.10250496119260788 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.12614209949970245 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.016270022839307785 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11362554877996445 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11290452629327774 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.971624493598938 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.15333333611488342 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01890118047595024 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.02931034564971924 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.14624017477035522 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.11622808128595352 *jet/accuracy_1_of_4‚0.6416661739349365 jet/accuracy_6_of_6‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.09113600105047226 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9345486164093018 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.11891891807317734 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.1975056529045105 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008920500054955482 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,particle/sensitivity‚0.23521868884563446 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.08828726410865784 particle/accuracy_2_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.07789269834756851 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17239493131637573 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9709302186965942 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 )loss/val_total_loss‚75.52442552300718 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.06019953638315201 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.09970725327730179 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9949758648872375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1111111119389534 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.024444444105029106 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.28270334005355835 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.11765748262405396 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.796444296836853 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.3106060326099396 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.021355442702770233 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.15894778072834015 jet/accuracy_5_of_6‚0 particle/accuracy_3_of_6‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.11353091150522232 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28178834915161133 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.015353364869952202 jet/accuracy_4_of_4‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03209083154797554 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9781050682067871 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.05795995518565178 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.04899231716990471 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9763419032096863 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.321908563375473 9 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.029263297095894814 +jet/accuracy_2_of_3‚0.09740963578224182 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0180533304810524 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9878336191177368 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12770108878612518 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028307797387242317 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9904624223709106 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 9 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.029347749426960945 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.5897344350814819 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.18284571170806885 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6025243401527405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 -jet/accuracy_3_of_3‚0.0054117245599627495 jet/accuracy_3_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9883027672767639 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.030741959810256958 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 *jet/accuracy_1_of_6‚0.6987180113792419 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.14633528888225555 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.015948602929711342 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.05000000074505806 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11874652653932571 particle/accuracy_2_of_4‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.13488969206809998 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.00204613106325268758d‡±E3¯ È£ & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.14887550473213196 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1101158931851387 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 : Purity/3FRt*SRqqt1FBt/FRt_purity‚0.0055555556900799274 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.09886793047189713 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3768160343170166 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚1098.425279119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9772205948829651 9 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.059950679540634155 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.26710575819015503 9 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.037748806178569794 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/3FRt3SRqqt*FBt/FRt_purity‚0 & Purity/1FRt*SRqqt3FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.02557678520679474 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1272840052843094 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2916208505630493 9 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.022972973063588142 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.11481481045484543 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.1666666567325592 9 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.030265051871538162 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.17901232838630676 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.0625 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.08812636137008667 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9941788911819458 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2271825224161148 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.03231654316186905 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.25738784670829773 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.03076357953250408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.040672462433576584 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.014880952425301075 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.12726861238479614 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9534366726875305 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.09723792970180511 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 +jet/accuracy_3_of_5‚0.01666666567325592 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.30701756477355957 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.11666666716337204 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.10815619677305222 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.07394636422395706 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7884413813758066e+09 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07121039927005768 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.26403144001960754 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.1465441882610321 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10335942357778549 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.002543752547353506 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05246913805603981 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.103050097823143 particle/accuracy_1_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.164198100566864 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.31874996423721313 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.962355375289917 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.15848568081855774 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.044312167912721634 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.30147767066955566 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03273829445242882 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.1274973452091217 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.25754597783088684 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9049707651138306 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9906757473945618 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.025531597435474396 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.08172816783189774 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 trainer/global_step‚1208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9774238467216492 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.05211000144481659 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33624330163002014 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3218509256839752 particle/accuracy_7_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/*FRt0SRqqt3FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.14561916887760162 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.013792130164802074 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581240415573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.05614035576581955 +jet/accuracy_1_of_2‚0.48445719480514526 particle/accuracy_3_of_5‚1 (particle/accuracy‚0.7881684899330139 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9742642045021057 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.04213183373212814 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.038018740713596344 particle/accuracy_3_of_3‚1 _step‚98 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.011005891487002373 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 jet/accuracy_1_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.04798850789666176 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.016071714460849762 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9287253618240356 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9913687109947205 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.013291110284626484 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.992583155632019 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03187708184123039 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9741886854171753 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.05652341619133949 ("Purity/2FRt2SRqqt1FBt/evP äcŠ^ent_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.06493055820465088 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.053614940494298935 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.06042885780334473 9 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.019086651504039764 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2652789354324341 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.18103446066379547 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846491813659668 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_4_of_5‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011066308245062828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9753362536430359 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.21759259700775146 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.5998294353485107 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9949758648872375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3496510684490204 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.036654695868492126 particle/accuracy_4_of_4‚1 jet/accuracy_4_of_6‚0 epoch‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FRt_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.1908131241798401 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.04545454680919647 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05268199369311333 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 +"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.11689426749944687 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9865273237228394 <"Purity/1FRt*SRqqt0FBt/event_purity‚0.0058403233997523785 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2645833492279053 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9561861753463745 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.07655038684606552 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9624934792518616 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0038548754528164864 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FRt_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2883397936820984 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15935011208057404 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.11689426749944687 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11303525418043137 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.039019130170345306 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.276717871427536 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.03533031791448593 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12284865975379944 +jet/accuracy_1_of_1‚0.27732330560684204 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9876915216445923 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.05321640148758888 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013785468181595206 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.11202432215213776 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1770833283662796 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.024154843762516975 & Purity/1FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.03556547686457634 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.013291110284626484 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.25816962122917175 & Purity/*FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887377619743347 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.02978750877082348 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.025757577270269394 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13772046566009521 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.033551864326000214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9941798448562622 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9898214340209961 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.08251488208770752 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2222016304731369 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24947331845760345 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9592934846878052 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.032568030059337616 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt3SRqqt0FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9953703880310059 particle/accuracy_5_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.034246791154146194 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.10250496119260788 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.12614209949970245 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.016270022839307785 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11362554877996445 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11290452629327774 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.971624493598938 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.15333333611488342 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01890118047595024 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.02931034564971924 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.14624017477035522 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.11622808128595352 *jet/accuracy_1_of_4‚0.6416661739349365 jet/accuracy_6_of_6‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.09113600105047226 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9345486164093018 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.11891891807317734 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.1975056529045105 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008920500054955482 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,particle/sensitivity‚0.23521868884563446 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.08828726410865784 particle/accuracy_2_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.07789269834756851 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17239493131637573 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9709302186965942 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 )loss/val_total_loss‚75.52442552300718 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.06019953638315201 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.09970725327730179 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9949758648872375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1111111119389534 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.024444444105029106 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.28270334005355835 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.11765748262405396 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.796444296836853 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.3106060326099396 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.021355442702770233 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.15894778072834015 jet/accuracy_5_of_6‚0 particle/accuracy_3_of_6‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.11353091150522232 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28178834915161133 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.015353364869952202 jet/accuracy_4_of_4‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03209083154797554 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9781050682067871 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.05795995518565178 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.04899231716990471 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9763419032096863 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.321908563375473 9 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.029263297095894814 +jet/accuracy_2_of_3‚0.09740963578224182 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0180533304810524 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9878336191177368 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12770108878612518 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028307797387242317 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9904624223709106 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 9 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.029347749426960945 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.5897344350814819 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.18284571170806885 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6025243401527405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 -jet/accuracy_3_of_3‚0.0054117245599627495 jet/accuracy_3_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9883027672767639 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.030741959810256958 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 *jet/accuracy_1_of_6‚0.6987180113792419 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.14633528888225555 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.015948602929711342 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.05000000074505806 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11874652653932571 particle/accuracy_2_of_4‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.13488969206809998 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FRt_purity‚0 9 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.016203705221414566 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.032393928617239 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0011691674590110779 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.10493826866149902 +jet/accuracy_2_of_2‚0.04096266254782677 jet/accuracy_5_of_5‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9737046957015991 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0004409171233419329 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.11206896603107452 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 particle/accuracy_6_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 ,jet/accuracy_3_of_6‚0.012820512987673283 particle/accuracy_4_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8904691934585571 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.03860034421086311 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.10876043885946274 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.02859094738960266 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.01383029855787754 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.00290291546843946 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9537158608436584 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.7317140102386475 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9366326332092285 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0011574075324460864 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.3106060326099396 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0bž>•œ° ‚(:‘ °æåÔ–¤ÃÇ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚190.55' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚38.11000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÏjC± ‚(:¸ °æåÔª«ªÊ network.recv‚447984$ disk.nvme0n1.out‚3882.1279296875! disk.dm-0.out‚3882.1279296875 disk.rbd1.out‚ 7.00390625 disk.sda.out‚ 40.26953125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.558241844177246 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0# memory_percent‚93.0389404296875 disk.nvme0c0n1.in‚0.078125 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 network.sent‚ 1.651712e+06 disk.sda2.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2712.90234375% disk./.usageGB‚291.43798446655273" proc.memory.availableMB‚1140.5 cpu‚25.31509423353445) disk./.usagePercent‚19.55864767030686& disk.nvme0c0n1.out‚3882.1279296875 disk.sda3.out‚ 40.26953125 disk.dm-1.out‚ 40.26953125 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 proc.cpu.threads‚17-«E]ž² ‚(:“ ¿æåÔÅúÜÇ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×TkÁJ³ ‚(:¿ ¿æåÔ¶ÖÓÊ* proc.memory.percent‚16.558241844177246 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1141.01171875# proc.memory.rssMB‚ 2712.90234375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 7.00390625 network.sent‚ 1.774802e+06 disk.sda2.out‚0 disk.sda3.out‚ 40.78515625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚454384 disk.drbd1716.out‚0$ memory_percent‚93.03581714630127% disk./.usageGB‚291.45117950439453 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 40.78515625 disk.nvme0n1.in‚0.078125! disk.dm-0.out‚5036.7021484375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 cpu‚25.31162516015854( disk./.usagePercent‚19.5595332003678 disk.sda.out‚ 40.78515625 disk.dm-1.in‚0$ disk.nvme0n1.out‚5036.7021484375 disk.drbd2080.in‚0& disk.nvme0c0n1.out‚5036.7021484375†é‡2´ - _wandbruntime‚1137  _runtime‚1137ˆÁ˜…ȵ  % _timestamp‚1.7884414201830351e+09 _runtime‚1137.230224747 _step‚99 ' lr-AdamW/pg1‚0.00015403499108843598 ' lr-AdamW/pg2‚0.00015403499108843598 trainer/global_step‚1249[\Ga̶ Æ ' lr-AdamW/pg1‚0.00015403499108843598 ' lr-AdamW/pg2‚0.00015403499108843598 trainer/global_step‚1249 % _timestamp‚1.7884414201830351e+09 _runtime‚1137.230224747 _step‚99còž„Í2· - _wandbruntime‚1137  _runtime‚1137ÕãUÆq¸ ÖO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98773269669888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88652853100032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27235236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97528721477632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87415500466079 _runtime‚1137.231213614 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1183894374400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89836730304000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88652835929600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19681534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210532352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49393456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6655524 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86235356009472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27235236 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87414702225963 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22135112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13025488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1179363255851 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27258344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1249 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89836730304000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1244758724608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 % _timestamp‚1.7884414201837974e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49370346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22135110 [KDeviceStatsMonitor.on_train_batch_stnb%kart/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98773480202240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97528696705024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1184677498880 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13025485 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1183894374400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1184677498880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86235338970112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1180144456607 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13025485 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1183894374400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1184677498880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86235338970112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1180144456607 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27235236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97528721477632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87415500466079 _runtime‚1137.231213614 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1183894374400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89836730304000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88652835929600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19681534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210532352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49393456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6655524 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86235356009472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27235236dÀè_2º - _wandbruntime‚1138  _runtime‚1138r›Kå&» ßM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200426496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49408720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22152152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1245532158464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚1249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87482443971295 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88721484673536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19696454 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89907089671168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89907089671168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1180287212255 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27279704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1138.034178372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87483242211619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1184821859840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86302156759040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13035480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788441420986404e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19695937 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6660974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225199104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13035477 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88721467503104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49431856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22152150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27279704 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88721484673536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1184821859840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97604403667968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89906289362944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27256570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22152150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0î!NÝ{¼ ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22152150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27279704 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88721484673536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1184821859840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97604403667968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89906289362944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27256570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22152150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200426496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49408720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22152152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1245532158464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚1249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87482443971295 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88721484673536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19696454 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89907089671168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89907089671168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1180287212255 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27279704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1138.034178372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87483242211619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1184821859840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86302156759040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13035480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788441420986404e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19695937 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6660974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225199104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13035477 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88721467503104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49431856e;Æ‚E2½ - _wandbruntime‚1138  _runtime‚1138‡„£P¾ Ê 1loss/FBt4/assignment_loss‚0.26564483857614263 /loss/FRt4/detection_loss‚1.9850937756773543 /loss/FBt2/detection_loss‚2.0752689114212988 /loss/FBt4/detection_loss‚1.8611559742136556 0loss/FBt2/assignment_loss‚0.2597074542218343 -loss/FRt1/detection_loss‚2.27161642263333 0loss/SRqqt3/detection_loss‚3.658210728317499 /loss/FRt1/assignment_loss‚5.656586956977844 2loss/SRqqt1/assignment_loss‚2.1894588418628858 /loss/FBt1/detection_loss‚6.1701692427907675 %loss/total_loss‚70.67683234103183 _step‚102 /loss/FRt3/assignment_loss‚6.388476079510104 0loss/FBt1/assignment_loss‚0.4787471443414688 .loss/FRt2/assignment_loss‚6.08483768546063 0loss/FBt3/assignment_loss‚0.6388264029201516 /loss/FRt2/detection_loss‚2.7405586087185405 0loss/SRqqt2/detection_loss‚4.228190962006064 1loss/SRqqt4/detection_loss‚1.1054163942202215 1loss/SRqqt3/assignment_loss‚2.618114377061526 0loss/SRqqt1/detection_loss‚2.821247143589932 trainer/global_step‚1249 % _timestamp‚1.7884414209880915e+09 _runtime‚1138.035240766 /loss/FRt4/assignment_loss‚4.699801885918395 -loss/FRt3/detection_loss‚3.27662423733742 0loss/SRqqt2/assignment_loss‚2.24072714106125 .loss/FBt3/detection_loss‚4.377267865526557 epoch‚1 2loss/SRqqt4/assignment_loss‚2.5850832666669574øm£¼T¿ Î 1loss/SRqqt3/assignment_loss‚2.618114377061526 0loss/SRqqt1/detection_loss‚2.821247143589932 trainer/global_step‚1249 % _timestamp‚1.7884414209880915e+09 _runtime‚1138.035240766 /loss/FRt4/assignment_loss‚4.699801885918395 -loss/FRt3/detection_loss‚3.27662423733742 0loss/SRqqt2/assignment_loss‚2.24072714106125 .loss/FBt3/detection_loss‚4.377267865526557 epoch‚1 2loss/SRqqt4/assignment_loss‚2.5850832666669574 1loss/FBt4/assignment_loss‚0.26564483857614263 /loss/FRt4/detection_loss‚1.9850937756773543 /loss/FBt2/detection_loss‚2.0752689114212988 /loss/FBt4/detection_loss‚1.8611559742136556 0loss/FBt2/assignment_loss‚0.2597074542218343 -loss/FRt1/detection_loss‚2.27161642263333 0loss/SRqqt3/detection_loss‚3.658210728317499 /loss/FRt1/assignment_loss‚5.656586956977844 2loss/SRqqt1/assignment_loss‚2.1894588418628858 /loss/FBt1/detection_loss‚6.1701692427907675 %loss/total_loss‚70.67683234103183 _step‚102 /loss/FRt3/assignment_loss‚6.388476079510104 0loss/FBt1/assignment_loss‚0.4787471443414688 .loss/FRt2/assignment_loss‚6.08483768546063 0loss/FBt3/assignment_loss‚0.6388264029201516 /loss/FRt2/detection_loss‚2.7405586087185405 0loss/SRqqt2/detection_loss‚4.228190962006064 1loss/SRqqt4/detection_loss‚1.1054163942202215f0ÜÝì“À ‚(:ˆ ÎæåÔÙâÈÈ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶—¸>Á ‚(:³ ÎæåÔ•®ÖÊ network.sent‚ 1.781099e+06$ disk./.usageGB‚291.4659957885742% disk.nvme0n1.out‚5096.38818359375 disk.dm-0.in‚0.078125* proc.memory.percent‚16.558241844177246 disk.sda1.out‚0) proc.memory.availableMB‚ 1139.01953125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0) disk./.usagePercent‚19.56052753363079 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚41.25 disk.sda3.in‚0 disk.sda3.out‚41.25 disk.drbd1716.in‚0" disk.dm-0.out‚5096.38818359375 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚5096.38818359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 7.3251953125 disk.rbd0.out‚0 disk.dm-1.out‚41.25# proc.memory.rssMB‚ 2712.90234375 cpu‚25.308473049050516 proc.cpu.threads‚17 network.recv‚459020 disk.rbd0.in‚0$ memory_percent‚93.04797649383545(uÇ>“ ‚(:ˆ ÝæåÔ¾ŸÔÇ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐFúKà ‚(:À ÝæåÔî¾Ê' disk.nvme0c0n1.out‚5149.50146484375 disk.dm-1.out‚ 42.1015625" disk.dm-0.out‚5149.50146484375 disk.sda3.in‚0$ memory_percent‚93.04981231689453& proc.memory.availableMB‚ 1138.71875 proc.cpu.threads‚17$ disk./.usageGB‚291.4769172668457 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 network.recv‚464467 disk.nvme0n1.in‚0.078125 cpu‚25.305198648699278 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0 network.sent‚ 1.888612e+06* disk./.usagePercent‚19.561260483201316 disk.drbd2080.out‚0 disk.sda.out‚ 42.1015625 disk.sda3.out‚ 42.1015625% disk.nvme0n1.out‚5149.50146484375# proc.memory.rssMB‚ 2712.90234375 disk.drbd2265.in‚0 disk.rbd1.out‚ 7.4111328125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.558241844177246 disk.sda1.in‚0 disk.sda1.out‚01m “Ä ‚(:ˆ ìæåÔå¡ØÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—lsþ=Å ‚(:² ìæåÔÌžèÉ disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 7.4736328125 disk.dm-1.out‚ 42.734375 disk.drbd2266.out‚0# disk.nvme0c0n1.out‚ 5220.8203125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125$ memory_percent‚93.05145740509033) proc.memory.availableMB‚ 1138.44921875) disk./.usagePercent‚19.56188002143418 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.30160958461624$ disk./.usageGB‚291.4861488342285! disk.nvme0n1.out‚ 5220.8203125 disk.dm-0.in‚0.078125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 42.734375 disk.sda3.out‚ 42.734375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚16.558241844177246 disk.sda.in‚0 network.recv‚469048 disk.dm-0.out‚ 5220.8203125# proc.memory.rssMB‚ 2712.90234375 network.sent‚ 1.894687e+06ÏÔ·2Æ - _wandbruntime‚1179  _runtime‚1179òõ´ÉÇ Ã ' lr-AdamW/pg1‚0.00015401412600648997 ' lr-AdamW/pg2‚0.00015401412600648997 trainer/global_step‚1299 % _timestamp‚1.7884414625715723e+09 _runtime‚1179.618773591 _step‚103Ô@Ã÷ÍÈ Ç % _timestamp‚1.7884414625715723e+09 _runtime‚1179.618773591 _step‚103 ' lr-AdamW/pg1‚0.00015401412600648997 ' lr-AdamW/pg2‚0.00015401412600648997 trainer/global_step‚1299g*£Y 2É - _wandbruntime‚1179  _runtime‚11795½òÞà'Ê ÚO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93314697665024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208828416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225574667259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6904632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90801803083771 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6904076 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22987110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1293263132672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93314697665024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101312806221824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51315508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89576228416512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93315497959936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92084431775232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89576245455872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101312830994432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28305288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20429161 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231066184704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22987110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28305288 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1226355868015 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20429720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233601024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90802601323887 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884414625722687e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92084414604800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22987112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1293054304256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1179.619750705 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28328396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521ã`åâä'Ë ÞO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102605860526080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13525085 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102606094127104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92084414604800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231066184704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1230283060224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51292398 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28328396 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13525088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92084431775232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93315497959936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1226355868015 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20429720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233601024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90802601323887 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884414625722687e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92084414604800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22987112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1293054304256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1179.619750705 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28328396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93314697665024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208828416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225574667259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6904632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90801803083771 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6904076 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22987110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1293263132672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93314697665024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101312806221824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51315508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89576228416512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93315497959936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92084431775232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89576245455872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101312830994432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28305288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20429161 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231066184704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22987110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28305288 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20444108 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1299 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101388488412160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1231993683456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1231210545664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28349756 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13535080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216810496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1294032651776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1231210545664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1231993683456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51353908 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23004150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6909575 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28326622 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101388488412160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1231993683456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1231210545664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28349756 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13535080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216810496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1294032651776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1231210545664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1231993683456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51353908 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23004150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6909575 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28326622 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20444108 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1299is‹†Ó2Ï - _wandbruntime‚1180  _runtime‚1180¬“‡ÄSÐ Í 1loss/SRqqt4/detection_loss‚0.9503828503657132 trainer/global_step‚1299 /loss/FRt1/assignment_loss‚5.992305456205856 /loss/FRt3/assignment_loss‚6.120703563094139 /loss/FRt2/detection_loss‚2.9258862231907092 .loss/FBt2/detection_loss‚2.285302462640833 %loss/total_loss‚68.20663913236118 1loss/SRqqt3/assignment_loss‚3.612645364943005 1loss/FBt3/assignment_loss‚0.44614014997949597 1loss/SRqqt2/detection_loss‚3.1830950329701104 1loss/SRqqt1/assignment_loss‚2.447488047786661 1loss/SRqqt1/detection_loss‚1.9198100200375996 /loss/FBt4/detection_loss‚2.3722885946190218 epoch‚1 .loss/FBt1/detection_loss‚4.018183863162994 1loss/SRqqt2/assignment_loss‚2.601444590759153 0loss/FBt1/assignment_loss‚0.7711411118507385 1loss/FBt2/assignment_loss‚0.17603961182279856 0loss/FBt4/assignment_loss‚0.3091937080897801 .loss/FRt1/detection_loss‚2.841016216160253 /loss/FRt4/detection_loss‚1.7904981456696987 1loss/SRqqt3/detection_loss‚3.8639615944453647 .loss/FBt3/detection_loss‚3.463907431066036 /loss/FRt2/assignment_loss‚6.041729299645675 /loss/FRt4/assignment_loss‚4.272049936652183 /loss/FRt3/detection_loss‚3.0837827660143375 _runtime‚1180.411483529 _step‚106 0loss/SRqqt4/assignment_loss‚2.71764309118901 $ _timestamp‚1.788441463364106e+09îÛ)WÑ Ñ .loss/FBt2/detection_loss‚2.285302462640833 %loss/total_loss‚68.20663913236118 1loss/SRqqt3/assignment_loss‚3.612645364943005 1loss/FBt3/assignment_loss‚0.44614014997949597 1loss/SRqqt2/detection_loss‚3.1830950329701104 1loss/SRqqt1/assignment_loss‚2.447488047786661 1loss/SRqqt1/detection_loss‚1.9198100200375996 /loss/FBt4/detection_loss‚2.3722885946190218 epoch‚1 .loss/FBt1/detection_loss‚4.018183863162994 1loss/SRqqt2/assignment_loss‚2.601444590759153 0loss/FBt1/assignment_loss‚0.7711411118507385 1loss/FBt2/assignment_loss‚0.17603961182279856 0loss/FBt4/assignment_loss‚0.3091937080897801 .loss/FRt1/detection_loss‚2.841016216160253 /loss/FRt4/detection_loss‚1.7904981456696987 1loss/SRqqt3/detection_loss‚3.8639615944453647 .loss/FBt3/detection_loss‚3.463907431066036 /loss/FRt2/assignment_loss‚6.041729299645675 /loss/FRt4/assignment_loss‚4.272049936652183 /loss/FRt3/detection_loss‚3.0837827660143375 _runtime‚1180.411483529 _step‚106 0loss/SRqqt4/assignment_loss‚2.71764309118901 $ _timestamp‚1.788441463364106e+09 1loss/SRqqt4/detection_loss‚0.9503828503657132 trainer/global_step‚1299 /loss/FRt1/assignment_loss‚5.992305456205856 /loss/FRt3/assignment_loss‚6.120703563094139 /loss/FRt2/detection_loss‚2.9258862231907092j9ÄCžÒ ‚(:“ ûæåÔ€¸ÕÇ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=A‡ð?Ó ‚(:´ ûæåÔ·ÚÊ disk.dm-0.out‚5282.630859375 disk.rbd1.out‚ 7.4736328125% disk.nvme0c0n1.out‚5282.630859375$ memory_percent‚93.05839538574219 cpu‚25.29866143621569 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 43.421875 proc.cpu.threads‚17 network.sent‚ 1.90057e+06 disk.drbd1716.in‚0 disk.drbd2266.out‚0 network.recv‚473098 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 43.421875% proc.memory.availableMB‚ 1137.3125# proc.memory.rssMB‚ 2712.90234375% disk./.usageGB‚291.50478744506836 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125# disk.nvme0n1.out‚5282.630859375 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 43.421875* proc.memory.percent‚16.558241844177246) disk./.usagePercent‚19.56313087424648䙧žÔ ‚(:“ ŠçåÔî߸Ç gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.551' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.910199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿzXIÕ ‚(:¾ ŠçåÔ¨—½Ê disk.dm-1.out‚ 44.0078125# memory_percent‚93.0596113204956 disk.nvme0n1.in‚0.078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda.out‚ 44.0078125 disk.sda3.out‚ 44.0078125) proc.memory.availableMB‚ 1137.11328125* proc.memory.percent‚16.558241844177246 cpu‚25.29583294279404 network.recv‚479710 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0& disk.nvme0c0n1.out‚5356.1259765625 disk.sda2.out‚0! disk.dm-0.out‚5356.1259765625 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 network.sent‚ 2.009018e+06* disk./.usagePercent‚19.564007956058784# proc.memory.rssMB‚ 2712.90234375$ disk.nvme0n1.out‚5356.1259765625 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.dm-1.in‚0$ disk./.usageGB‚291.5178565979004 disk.rbd1.out‚ 8.2119140625g#qÌ’Ö ‚(:‡ ™çåÔß¼–Ç gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ñ—`L× ‚(:Á ™çåÔ«üÊ) proc.memory.availableMB‚ 1137.08203125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.recv‚482998( disk./.usagePercent‚19.5654884476293 disk.drbd1716.out‚0 disk.drbd2265.out‚0$ disk.nvme0n1.out‚5411.6064453125 disk.drbd2265.in‚0 disk.rbd1.out‚ 8.3134765625 network.sent‚ 2.014353e+06 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 44.50390625 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.558241844177246 disk.nvme0c0n1.in‚0.078125 disk.dm-1.out‚ 44.50390625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚5411.6064453125 disk.sda2.out‚0 disk.dm-0.in‚0.078125! disk.dm-0.out‚5411.6064453125 disk.rbd1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚291.5399169921875 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2712.90234375 cpu‚25.292670567356357 disk.sda3.out‚ 44.50390625 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚93.05980205535889Írtu2Ø - _wandbruntime‚1222  _runtime‚1222³7ØËÈÙ Â ' lr-AdamW/pg1‚0.00015397721547736048 ' lr-AdamW/pg2‚0.00015397721547736048 trainer/global_step‚1349 $ _timestamp‚1.788441505514088e+09 _runtime‚1222.561297234 _step‚107VT,ÌÚ Æ $ _timestamp‚1.788441505514088e+09 _runtime‚1222.561297234 _step‚107 ' lr-AdamW/pg1‚0.00015397721547736048 ' lr-AdamW/pg2‚0.00015397721547736048 trainer/global_step‚1349k¹í?2Û - _wandbruntime‚1222  _runtime‚1222 ž'Ü ÙO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95515993280000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚ 1222.56221231 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29375340 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚108 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1271786078667 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14024688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106438775895040 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96792665026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23839112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96792665026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23839110 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1277454870528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94189702181695 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95515993280000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1277454870528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105096940511232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 trainer/global_step‚1349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14024685 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95516010450432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884415055147831e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96793465320960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29375340 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1276671746048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23839112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23839110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92917117862912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272567279423 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96793465320960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7154439 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0gôØxã'Ý ÝO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92917134902272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94188903941579 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1276671746048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105096915738624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884415055147831e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96793465320960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29375340 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1276671746048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23839112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23839110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92917117862912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272567279423 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96793465320960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7154439 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95515993280000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚ 1222.56221231 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29375340 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚108 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1271786078667 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14024688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106438775895040 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96792665026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23839112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96792665026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23839110 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1277454870528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94189702181695 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95515993280000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1277454870528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105096940511232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 trainer/global_step‚1349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14024685 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95516010450432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96862224084992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106515651369472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23856150 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23856152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95584642023936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7159449 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273491236035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1342845018624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21193592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96863024393216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53252824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1277599231488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53275960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23856152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14034677 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105172622701568 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257443927235 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95584642023936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183649280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256645686911 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21194129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92983935651840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29419808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29396674 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53275960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106515442947584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1272710035071_¡‹d#à æM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95584624853504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1278382369280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105172597928960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92983952691200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53252824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95584624853504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14034680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1343028667904 _step‚109 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96863024393216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚ 1223.37833886 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95584642023936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7159449 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273491236035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1342845018624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21193592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96863024393216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53252824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1277599231488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53275960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23856152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14034677 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105172622701568 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257443927235 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95584642023936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183649280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256645686911 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21194129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92983935651840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29419808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29396674 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53275960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106515442947584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1272710035071 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1278382369280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29396674 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96862224084992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106515651369472 E=DeviceStatsMonitor.Šã9pÚon_train_batch_end/segment.large_pool.peak‚521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23856150 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23856152 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30445392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317997490075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98947571955200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96258007309312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98947589125632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30445392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1323843556352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7403744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97576004799387 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98947589125632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14524288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100270632387072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108881050028032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1390929330176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1390745667584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100270632387072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97576803039503 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96258024348672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚1399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7403238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884415479700885e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110271770923008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30468500 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110271979358208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1318778690831 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚112Ïèç'ï áO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98947589125632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30445392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1323843556352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7403744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97576004799387 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98947589125632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14524288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100270632387072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108881050028032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1390929330176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1390745667584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100270632387072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97576803039503 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96258024348672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚1399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7403238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884415479700885e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110271770923008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30468500 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110271979358208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1318778690831 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1323843556352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30468500 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14524285 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 _runtime‚ 1265.01749981 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24691112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100271432681984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1323060431872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98947571955200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21928032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108881025255424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55136502 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24691110 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55159612 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30445392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317997490075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98947571955200 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96258007309312pçS(72ð - _wandbruntime‚1265  _runtime‚1265iTt6 ñ çM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚99016220699136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55198012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21942501 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14534280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788441548760182e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99016203528704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108956732218368 VGDeviceStatsMonitor.on_train_batch_end/rese楷rved_bytes.small_pool.current‚ 1025507328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99016203528704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30466726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55174876 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55174876 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7408224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214713344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96324825098240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24708152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55198012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21942501 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14534280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788441548760182e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99016203528704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108956732218368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99016203528704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30466726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55174876 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55174876 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7408224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214713344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96324825098240q“$±}2ó - _wandbruntime‚1265  _runtime‚1265ˆî¾Sô Í 2loss/SRqqt4/assignment_loss‚2.6869606797893844 .loss/FBt3/detection_loss‚3.779929152556828 -loss/FBt4/detection_loss‚8.16816582242326 trainer/global_step‚1399 0loss/FRt4/assignment_loss‚4.8282347066061835 2loss/SRqqt1/assignment_loss‚2.6968752551078796 1loss/SRqqt2/assignment_loss‚3.212266855769687 2loss/SRqqt3/assignment_loss‚3.0148435008936914 /loss/FRt3/detection_loss‚3.8830051918824515 epoch‚1 0loss/FBt3/assignment_loss‚0.7125510213850897 -loss/FBt1/detection_loss‚5.13368968963623 % _timestamp‚1.7884415487617252e+09 _runtime‚1265.808850057 _step‚114 /loss/FRt1/assignment_loss‚5.991375429289682 /loss/FRt2/assignment_loss‚5.789657990137736 /loss/FBt1/assignment_loss‚0.487912979722023 1loss/SRqqt2/detection_loss‚3.3993125193648868 %loss/total_loss‚75.96747791556047 0loss/FBt2/assignment_loss‚0.4132015106727704 /loss/FRt3/assignment_loss‚6.796214421590169 0loss/FBt4/assignment_loss‚0.5955867900305011 0loss/SRqqt1/detection_loss‚2.408980284333229 1loss/SRqqt4/detection_loss‚1.0010013598327836 /loss/FRt1/detection_loss‚2.2174537080739225 /loss/FBt2/detection_loss‚1.8235075123608113 /loss/FRt2/detection_loss‚2.4390194912751517 /loss/FRt4/detection_loss‚1.6624932723740737 0loss/SRqqt3/detection_loss‚2.825238770452039°’›Wõ Ñ 0loss/FRt4/assignment_loss‚4.8282347066061835 2loss/SRqqt1/assignment_loss‚2.6968752551078796 1loss/SRqqt2/assignment_loss‚3.212266855769687 2loss/SRqqt3/assignment_loss‚3.0148435008936914 /loss/FRt3/detection_loss‚3.8830051918824515 epoch‚1 0loss/FBt3/assignment_loss‚0.7125510213850897 -loss/FBt1/detection_loss‚5.13368968963623 % _timestamp‚1.7884415487617252e+09 _runtime‚1265.808850057 _step‚114 /loss/FRt1/assignment_loss‚5.991375429289682 /loss/FRt2/assignment_loss‚5.789657990137736 /loss/FBt1/assignment_loss‚0.487912979722023 1loss/SRqqt2/detection_loss‚3.3993125193648868 %loss/total_loss‚75.96747791556047 0loss/FBt2/assignment_loss‚0.4132015106727704 /loss/FRt3/assignment_loss‚6.796214421590169 0loss/FBt4/assignment_loss‚0.5955867900305011 0loss/SRqqt1/detection_loss‚2.408980284333229 1loss/SRqqt4/detection_loss‚1.0010013598327836 /loss/FRt1/detection_loss‚2.2174537080739225 /loss/FBt2/detection_loss‚1.8235075123608113 /loss/FRt2/detection_loss‚2.4390194912751517 /loss/FRt4/detection_loss‚1.6624932723740737 0loss/SRqqt3/detection_loss‚2.825238770452039 2loss/SRqqt4/assignment_loss‚2.6869606797893844 .loss/FBt3/detection_loss‚3.779929152556828 -loss/FBt4/detection_loss‚8.16816582242326 trainer/global_step‚1399rÛ… žö ‚(:“ ÕçåÔ»ÇšÇ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.548' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.309599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 >~D÷ ‚(:¹ ÕçåÔ˜ˆšÊ! disk.dm-0.out‚5640.8916015625 disk.rbd1.out‚ 9.142578125 disk.nvme0c0n1.in‚0.078125& disk.nvme0c0n1.out‚5640.8916015625# proc.memory.rssMB‚ 2712.90234375 disk.sda.out‚ 46.890625 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 network.sent‚ 2.139871e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.09775257110596 disk.dm-1.out‚ 46.890625 disk.drbd2265.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 147.82421875 cpu‚25.28128899514359# disk./.usageGB‚291.553768157959 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.recv‚501189 disk.sda2.in‚0$ disk.nvme0n1.out‚5640.8916015625 disk.drbd2266.in‚0* proc.memory.percent‚16.558241844177246* disk./.usagePercent‚19.566418010986123 disk.sda3.in‚0 disk.sda3.out‚ 46.890625 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd1716.out‚0ä%=Uø ‚(:… äçåÔïü¦Ç gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[é”Mù ‚(: äçåÔñÖ¾É cpu‚25.2788708920706 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd1.out‚ 9.205078125 disk.sda1.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 147.67578125* proc.memory.percent‚16.558241844177246$ disk./.usageGB‚291.5678405761719 disk.dm-0.in‚0.078125 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚507753 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚5694.27197265625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 47.41015625# proc.memory.rssMB‚ 2712.90234375 disk.drbd2080.out‚0$ memory_percent‚99.09865856170654* disk./.usagePercent‚19.567362422779937 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 47.41015625 disk.dm-1.out‚ 47.41015625 network.sent‚ 2.248268e+06% disk.nvme0n1.out‚5694.27197265625' disk.nvme0c0n1.out‚5694.27197265625 disk.sda2.out‚0 ë˜6‘ú ‚(:† óçåÔ»¼¤Ç gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(†ÅOû ‚(:Ä óçåÔºö·Ê% disk./.usageGB‚291.57415771484375 disk.sda.out‚ 48.16015625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1716.in‚0" disk.dm-0.out‚5736.16650390625 disk.rbd0.in‚0* proc.memory.percent‚16.558241844177246 network.recv‚511779 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2712.90234375 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚5736.16650390625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.254061e+06 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚5736.16650390625 disk.sda2.out‚0 disk.dm-1.out‚ 48.16015625 disk.drbd1716.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 148.26171875* disk./.usagePercent‚19.567786371256638 disk.sda3.out‚ 48.16015625 disk.drbd2266.in‚0$ memory_percent‚99.09508228302002 cpu‚25.27606676305537 disk.sda.in‚0 disk.rbd1.out‚ 9.470703125a3qT2ü - _wandbruntime‚1307  _runtime‚1307&ÒlðÈý  _step‚115 ' lr-AdamW/pg1‚0.00015385529218819096 ' lr-AdamW/pg2‚0.00015385529218819096 trainer/global_step‚1449 % _timestamp‚1.7884415904083886e+09 _runtime‚ 1307.45565279%<.Ìþ Æ ' lr-AdamW/pg2‚0.00015385529218819096 trainer/global_step‚1449 % _timestamp‚1.7884415904083886e+09 _runtime‚ 1307.45565279 _step‚115 ' lr-AdamW/pg1‚0.00015385529218819096sDè2ÿ - _wandbruntime‚1307  _runtime‚1307˜îUu €äO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103749400043008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99598896755712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25543112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1370232242176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1364208901483 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100963105657195 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 _step‚116 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25543112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57081664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114104972222464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1369449117696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100963903897311 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99598913795072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884415904091053e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102379150630400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1370232242176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114104765884416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22676315 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103749400043008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103748599748096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112665134772224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103748599748096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31515444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181565440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31538552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1369449117696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22675797 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439812677632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102379150630400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112665159544832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31538552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206338048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102379167800832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25543110 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚1307.456554341 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31538552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206338048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102379167800832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25543110 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚1307.456554341 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99598896755712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25543112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1370232242176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1364208901483 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100963105657195 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 _step‚116 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25543112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57081664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114104972222464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1369449117696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100963903897311 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99598913795072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884415904091053e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102379150630400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1370232242176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114104765884416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22676315 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103749400043008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103748599748096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112665134772224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103748599748096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31515444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181565440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31538552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1369449117696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22675797 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439812677632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102379150630400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112665159544832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99665714544640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1365914058851 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1365132857887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102447782203904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1371159740928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1370376603136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57120064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102447782203904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198329344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57120064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22690795 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7656918 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1370376603136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31536778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22691316 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31536778 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57096928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103818959115264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102447799374336 _step‚117 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102447799374336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99665731584000 _Gø&„òM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102447799374336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99665731584000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1371159740928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440791220736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31559912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7657436 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25560150 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114181409853952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112740841735168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114181632955904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚101031645642851 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103818158807040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚1449 _runtime‚1308.257087021 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440592891392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15033877 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99665714544640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1365914058851 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1365132857887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102447782203904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1371159740928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1370376603136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57120064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102447782203904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198329344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57120064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22690795 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7656918 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1370376603136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31536778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22691316 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31536778 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57096928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103818959115264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102447799374336 _step‚117uI0á­2…- _wandbruntime‚1308  _runtime‚1308 xƒ†Ì 0loss/FBt3/assignment_loss‚0.5887010637492164 /loss/FBt2/detection_loss‚2.8120337981908095 -loss/FBt4/detection_loss‚1.97287029254745 _step‚118 /loss/FRt4/assignment_loss‚4.503483006983627 2loss/SRqqt2/assignment_loss‚2.6615017311913625 1loss/SRqqt1/detection_loss‚2.1252466116921376 .loss/FBt1/detection_loss‚4.870515012741089 .loss/FBt3/detection_loss‚6.423125380819494 1loss/SRqqt4/assignment_loss‚2.601004823616573 /loss/SRqqt3/detection_loss‚3.37814924120903 % _timestamp‚1.788441î–-65912111197e+09 .loss/FRt3/assignment_loss‚6.27334713113719 .loss/FRt2/detection_loss‚2.404670923948288 /loss/FRt4/detection_loss‚1.8579964646954594 epoch‚1 0loss/FBt1/assignment_loss‚0.7596747111529112 _runtime‚1308.258250778 2loss/SRqqt3/assignment_loss‚3.4104965262942843 /loss/FRt3/detection_loss‚3.0807233308923654 trainer/global_step‚1449 /loss/FRt1/detection_loss‚2.4180403417736596 /loss/FRt1/assignment_loss‚6.002636422916335 /loss/FRt2/assignment_loss‚6.068282276391983 1loss/SRqqt1/assignment_loss‚2.486657412375434 1loss/FBt2/assignment_loss‚0.34193463738961327 0loss/FBt4/assignment_loss‚0.2673151273844728 1loss/SRqqt2/detection_loss‚3.1965584471112205 1loss/SRqqt4/detection_loss‚1.1412380657291838 %loss/total_loss‚71.64620278193318àuòV‡Ð /loss/FBt2/detection_loss‚2.8120337981908095 -loss/FBt4/detection_loss‚1.97287029254745 _step‚118 /loss/FRt4/assignment_loss‚4.503483006983627 2loss/SRqqt2/assignment_loss‚2.6615017311913625 1loss/SRqqt1/detection_loss‚2.1252466116921376 .loss/FBt1/detection_loss‚4.870515012741089 .loss/FBt3/detection_loss‚6.423125380819494 1loss/SRqqt4/assignment_loss‚2.601004823616573 /loss/SRqqt3/detection_loss‚3.37814924120903 % _timestamp‚1.7884415912111197e+09 .loss/FRt3/assignment_loss‚6.27334713113719 .loss/FRt2/detection_loss‚2.404670923948288 /loss/FRt4/detection_loss‚1.8579964646954594 epoch‚1 0loss/FBt1/assignment_loss‚0.7596747111529112 _runtime‚1308.258250778 2loss/SRqqt3/assignment_loss‚3.4104965262942843 /loss/FRt3/detection_loss‚3.0807233308923654 trainer/global_step‚1449 /loss/FRt1/detection_loss‚2.4180403417736596 /loss/FRt1/assignment_loss‚6.002636422916335 /loss/FRt2/assignment_loss‚6.068282276391983 1loss/SRqqt1/assignment_loss‚2.486657412375434 1loss/FBt2/assignment_loss‚0.34193463738961327 0loss/FBt4/assignment_loss‚0.2673151273844728 1loss/SRqqt2/detection_loss‚3.1965584471112205 1loss/SRqqt4/detection_loss‚1.1412380657291838 %loss/total_loss‚71.64620278193318 0loss/FBt3/assignment_loss‚0.5887010637492164vùÄ-œˆ‚(:‘ ‚èåÔîôÅÇ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚195.1' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.019999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~m~YO‰‚(:Ä ‚èåÔÔ›…Ê disk.sda.out‚ 48.68359375 disk.drbd2266.out‚0 disk.dm-0.in‚0.078125 disk.sda1.out‚0$ memory_percent‚99.09548759460449 disk.rbd0.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚5772.57080078125 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.sent‚ 2.260463e+06" disk.dm-0.out‚5772.57080078125 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125* proc.memory.percent‚16.558241844177246 network.recv‚516657% disk./.usageGB‚291.59450912475586 disk.dm-1.out‚ 48.68359375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2712.90234375 disk.drbd2265.in‚0 disk.sda3.out‚ 48.68359375' proc.memory.availableMB‚ 148.1953125 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 9.658203125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚5772.57080078125* disk./.usagePercent‚19.569152171451808 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 cpu‚25.273376923769995&§|Š“Š‚(:ˆ ‘èåÔ×±ÊÇ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÇÄ’O‹‚(:Ä ‘èåÔÏü¹É* disk./.usagePercent‚19.571027682647653 disk.dm-1.in‚0* proc.memory.percent‚16.558241844177246$ disk./.usageGB‚291.6224555969238 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 network.recv‚522264 disk.drbd2080.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5823.49267578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 49.05859375 disk.drbd1716.out‚0 network.sent‚ 2.367904e+06 disk.sda3.in‚0$ memory_percent‚99.09703731536865 cpu‚25.271121602001532 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 disk.rbd1.out‚ 9.658203125 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0( proc.memory.availableMB‚ 147.94140625# proc.memory.rssMB‚ 2712.90234375 disk.rbd1.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚5823.49267578125 disk.dm-0.in‚0.078125" disk.dm-0.out‚5823.49267578125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda.out‚ 49.05859375 disk.dm-1.out‚ 49.05859375+¸«ç2Œ- _wandbruntime‚1349  _runtime‚1349ï½ØÈ trainer/global_step‚1499 $ _timestamp‚1.788441632713102e+09 _runtime‚1349.760320254 _step‚119 ' lr-AdamW/pg1‚0.00015377030483678164 ' lr-AdamW/pg2‚0.00015377030483678164»ßSµÌŽÆ $ _timestamp‚1.788441632713102e+09 _runtime‚1349.760320254 _step‚119 ' lr-AdamW/pg1‚0.00015377030483678164 ' lr-AdamW/pg2‚0.00015377030483678164 trainer/global_step‚1499w_Ï2­2- _wandbruntime‚1349  _runtime‚13498Åÿë'åO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107226567109120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105810746476032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411201513647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116449269061632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107227367404032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59003716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117937843136000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104351004755119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200439808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58980606 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416620928000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26395112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488574074368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1415837803520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1488373634560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32585496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416620928000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117937617923584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102939803241472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23426261 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚1499 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15523488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225212416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102939786202112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105810746476032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107226567109120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7902773 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104350206515003 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0³— `ï'‘éO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107227367404032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59003716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117937843136000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104351004755119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200439808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58980606 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416620928000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26395112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488574074368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1415837803520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1488373634560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32585496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416620928000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117937617923584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102939803241472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23426261 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚1499 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15523488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225212416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102939786202112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105810746476032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107226567109120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7902773 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104350206515003 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107226567109120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105810746476032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411201513647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116449269061632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489xY©ð“’‚(:ˆ  èåÔ‡òÛÇ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.297' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªѧP“‚(:Å  èåÔ¦ß×É disk.sda3.out‚ 49.53515625* proc.memory.percent‚16.558241844177246 network.recv‚527060" disk.dm-0.out‚5879.81982421875 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2712.90234375 proc.cpu.threads‚17 disk.sda.out‚ 49.53515625* disk./.usagePercent‚19.572356105742827 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125$ memory_percent‚99.09548759460449 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚25.268520594528894 disk.rbd1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚5879.81982421875 network.sent‚ 2.374048e+06% disk./.usageGB‚291.64225006103516 disk.drbd1716.in‚0% disk.nvme0n1.out‚5879.81982421875 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 9.6904296875 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 49.53515625 disk.nvme0n1.in‚0.078125 disk.sda1.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 148.1953125 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda.in‚0ÇîRÙ2”- _wandbruntime‚1350  _runtime‚1350¤NìP•ðM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59018980 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105879360879104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1416765288960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116524951251968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412125470259 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32606830 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105879360879104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23441165 _runtime‚1350.560506451 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23440646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚121 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1489539671040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15533477 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116524926479360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103006603991040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104417948260335 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32629964 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7907169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411344269295 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884416335125139e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417548426752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59042116 C9DeviceStatsMonitoiÀ-%¦r.on_train_batch_end/allocation.all.peak‚28215 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214713344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103006621030400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59042116 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1417548426752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26412152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26412150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1416765288960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26412152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26412150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105879378049536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107296126168064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15533480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26412150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1416765288960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26412152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26412150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105879378049536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107296126168064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15533480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1416765288960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116524951251968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412125470259 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32606830 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105879360879104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23441165 _runtime‚1350.560506451 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23440646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚121 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1489539671040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15533477 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116524926479360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103006603991040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104417948260335 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32629964 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7907169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411344269295 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884416335125139e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417548426752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59042116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214713344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103006621030400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59042116 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1417548426752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26412152 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33678656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120233353805824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109242307980800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60925768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33655548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8152969 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109242307980800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109242325151232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16023088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8153457 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1537430840832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110705334765056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 _runtime‚ 1392.08539467 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33678656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60902658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1537247178240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107738105612927 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106280692687872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1457412925055 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27247112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121770809419264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1463009613824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109242325151232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107737307372811 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110705334765056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110704534470144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1463009613824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27247110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109242325151232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107737307372811 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110705334765056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110704534470144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1463009613824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27247110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33678656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120233353805824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109242307980800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60925768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33655548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8152969 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0|`Ù2¤- _wandbruntime‚1392  _runtime‚1392a/k‚õ&¥ïM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1457555680703 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60964168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24191067 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27264150 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109310956724736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463153974784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121847475458048 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121847254453248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33700016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120309060768768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16033080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33676882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109310939554304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109310956724736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110774093529088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33676882 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16033077 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60964168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1463937112576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27264150 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27264152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60941032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1538414689280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120309035996160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106347493437440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110774893837312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¶.ù&¦óM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24191566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107805847358467 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107805049118143 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110774893837312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1457555680703 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60964168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24191067 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27264150 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109310956724736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463153974784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121847475458048 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121847254453248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33700016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120309060768768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16033080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33676882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109310939554304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109310956724736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110774093529088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33676882 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16033077 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60964168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1463937112576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27264150 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27264152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60941032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1538414689280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120309035996160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106347493437440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EB+“®‚(:ˆ ëèåÔýßíÇ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.019' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ç!¥J¯‚(:¿ ëèåÔ‰ù¾É network.recv‚552651 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚6125.07666015625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 9.99609375 disk.rbd0.in‚0 disk.sda3.out‚ 52.2734375 disk.drbd1716.in‚0$ memory_percent‚99.10008907318115 cpu‚25.257233367256923 disk.dm-1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 147.44140625 disk.dm-1.out‚ 52.2734375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2712.90234375 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.558241844177246 disk.sda.in‚0% disk.nvme0n1.out‚6125.07666015625 network.sent‚ 2.687133e+06 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125) disk./.usagePercent‚19.57740739040092$ disk./.usageGB‚291.7175178527832 disk.rbd1.in‚0 disk.sda.out‚ 52.2734375 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚6125.07666015625 proc.cpu.threads‚176CÞê2°- _wandbruntime‚1434  _runtime‚1434o×[’ɱà ' lr-AdamW/pg1‚0.00015355236727280627 ' lr-AdamW/pg2‚0.00015355236727280627 trainer/global_step‚1599 % _timestamp‚1.7884417175106971e+09 _runtime‚1434.557905744 _step‚127Á 78Í²Ç ' lr-AdamW/pg1‚0.00015355236727280627 ' lr-AdamW/pg2‚0.00015355236727280627 trainer/global_step‚1599 % _timestamp‚1.7884417175106971e+09 _runtime‚1434.557905744 _step‚127B4G 2³- _wandbruntime‚1434  _runtime‚1434ÛèHÙë'´åO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114183302126080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112673903826432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28099112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1503624336463 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8404859 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24927544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1502843135707 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28099110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114183302126080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16522688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109621565094912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚1434.558826381 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114182501831168 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34725600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 _step‚128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788441717511392e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1508615175168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28099112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109621582134272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16522685 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1508615175168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1509398299648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125603872562176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112673886656000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112673903826432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34725600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34748708 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1509398299648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62847820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8405386 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212629504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125603659932672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124017463322624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124017488095232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187856896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62824710 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24928074 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34748708 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111125206470735 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114182501831168 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114182501831168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114183302126080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112673903826432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28099112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1503624336463 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8404859 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24927544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1502843135707 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28099110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114183302126080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16522688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109621565094912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚1434.558826381 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114182501831168 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34725600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 _step‚128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788441717511392e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1508615175168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28099112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109621582134272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16522685 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1508615175168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1509398299648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125603872562176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112673886656000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112673903826432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34725600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34748708 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1509398299648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62847820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8405386 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212629504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125603659932672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124017463322624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1€¦Œm2¶- _wandbruntime‚1435  _runtime‚1435Vì·ðM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124093145512960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28116150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16532677 _step‚129 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34746934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8410330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124093170285568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114252861198336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24942483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111192149975951 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1504548293075 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1510325798400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112742535399936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62863084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125680543359488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34770068 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1587373073920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34770068 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114252861198336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1509542660608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1509542660608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28116152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34746934 trÇ }3wainer/global_step‚1599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114252060890112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1503767092111 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28116152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62886220 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28116150 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62863084 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1510325798400Þä¢Bû&¸õM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62886220 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28116150 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62863084 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1510325798400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124093145512960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28116150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16532677 _step‚129 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34746934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8410330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124093170285568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114252861198336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24942483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111192149975951 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1504548293075 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1510325798400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112742535399936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62863084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125680543359488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34770068 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1587373073920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34770068 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114252861198336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1509542660608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1509542660608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28116152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34746934 trainer/global_step‚1599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114252060890112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1503767092111 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28116152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ë Ö2¹- _wandbruntime‚1435  _runtime‚1435f ÿUºÏ trainer/global_step‚1599 $ _timestamp‚1.788441718324851e+09 1loss/SRqqt1/detection_loss‚2.4398219977937092 /loss/FBt2/detection_loss‚0.8544134931342507 .loss/FBt3/detection_loss‚4.410334275319026 epoch‚1 1loss/FBt2/assignment_loss‚0.25104752920151957 .loss/FBt4/detection_loss‚3.100507599115372 %loss/total_loss‚66.88380585720364 /loss/FRt4/assignment_loss‚3.672189474105835 0loss/SRqqt1/assignment_loss‚2.65846013514007 /loss/FRt1/detection_loss‚2.7419722828923203 /loss/FRt4/detection_loss‚2.0986770630859937 /loss/FBt1/detection_loss‚4.3717504978179935 /loss/FRt1/assignment_loss‚5.616879416675102 /loss/FRt3/assignment_loss‚5.980168972015381 2loss/SRqqt2/assignment_loss‚2.4054569063278346 2loss/SRqqt3/assignment_loss‚3.0501061975955963 0loss/FBt4/assignment_loss‚0.6141792535050626 0loss/FBt1/assignment_loss‚0.3668747365474701 1loss/SRqqt2/detection_loss‚3.4972323912840624 1loss/SRqqt3/detection_loss‚3.0744402781128883 .loss/FRt2/assignment_loss‚4.29862089628397 2loss/SRqqt4/assignment_loss‚2.4146626458718226 0loss/FBt3/assignment_loss‚0.5984438204188616 .loss/FRt2/detection_loss‚4.228942167447057 /loss/FRt3/detection_loss‚3.1878783774375914 1loss/SRqqt4/detection_loss‚0.9507454500748561 _runtime‚1435.371987546 _step‚130®å×Z»Ô 1loss/SRqqt1/detection_loss‚2.4398219977937092 /loss/FBt2/detection_loss‚0.8544134931342507 .loss/FBt3/detection_loss‚4.410334275319026 epoch‚1 1loss/FBt2/assignment_loss‚0.25104752920151957 .loss/FBt4/detection_loss‚3.100507599115372 %loss/total_loss‚66.88380585720364 /loss/FRt4/assignment_loss‚3.672189474105835 0loss/SRqqt1/assignment_loss‚2.65846013514007 /loss/FRt1/detection_loss‚2.7419722828923203 /loss/FRt4/detection_loss‚2.0986770630859937 /loss/FBt1/detection_loss‚4.3717504978179935 /loss/FRt1/assignment_loss‚5.616879416675102 /loss/FRt3/assignment_loss‚5.980168972015381 2loss/SRqqt2/assignment_loss‚2.4054569063278346 2loss/SRqqt3/assignment_loss‚3.0501061975955963 0loss/FBt4/assignment_loss‚0.6141792535050626 0loss/FBt1/assignment_loss‚0.3668747365474701 1loss/SRqqt2/detection_loss‚3.4972323912840624 1loss/SRqqt3/detection_loss‚3.0744402781128883 .loss/FRt2/assignment_loss‚4.29862089628397 2loss/SRqqt4/assignment_loss‚2.4146626458718226 0loss/FBt3/assignment_loss‚0.5984438204188616 .loss/FRt2/detection_loss‚4.228942167447057 /loss/FRt3/detection_loss‚3.1878783774375914 1loss/SRqqt4/detection_loss‚0.9507454500748561 _runtime‚1435.371987546 _step‚130 trainer/global_step‚1599 $ _timestamp‚1.788441718324851e+09‚seeÄ“¼‚(:ˆ úèåÔð×Ç gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.899' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JDä5½‚(:ª úèåÔ“ÝçÉ disk.nvme0n1.in‚0.078125 disk.dm-0.out‚6182.375 disk.sda.in‚0 disk.nvme0n1.out‚6182.375 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 52.74609375 network.sent‚ 2.693401e+06 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0$ memory_percent‚99.10039901733398* proc.memory.percent‚16.558241844177246 cpu‚25.25493126032394 disk.rbd1.in‚0 disk.sda1.in‚0* disk./.usagePercent‚19.578841544607737 disk.sda.out‚ 52.74609375 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0& proc.memory.availableMB‚ 147.390625% disk./.usageGB‚291.73888778686523 disk.rbd1.out‚ 10.36328125 disk.sda1.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2712.90234375 disk.sda3.out‚ 52.74609375 disk.nvme0c0n1.in‚0.078125 disk.nvme0c0n1.out‚6182.375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 network.recv‚557257 disk.sda2.in‚0 disk.sda2.out‚0žX®ð¾‚(:’ ‰éåÔ²òÇ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚140.42' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.083999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LšLÿB¿‚(:· ‰éåÔîÔûÉ disk.sda.in‚0 disk.drbd2266.out‚0 network.recv‚562594 disk.sda2.out‚0 network.sent‚ 2.800855e+06 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0# memory_percent‚99.1004467010498 cpu‚25.252850569064645 proc.cpu.threads‚17% disk./.usageGB‚291.75413513183594 disk.sda1.in‚0 disk.sda3.out‚ 53.31640625# proc.memory.rssMB‚ 2712.90234375 disk.dm-1.in‚0 disk.dm-1.out‚ 53.31640625 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 147.3828125 disk.dm-0.out‚ 6229.4609375 disk.drbd2266.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.558241844177246* disk./.usagePercent‚19.579864806722096! disk.nvme0n1.out‚ 6229.4609375 disk.drbd2080.out‚0 disk.rbd1.out‚ 10.36328125 disk.sda.out‚ 53.31640625 disk.sda3.in‚0# disk.nvme0c0n1.out‚ 6229.4609375(pÌû“À‚(:ˆ ˜éåÔ–§ÊÇ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò«jmNÁ‚(:à ˜éåÔâãÉ* disk./.usagePercent‚19.581159948837815" disk.dm-0.out‚6294.67236328125 disk.drbd2265.out‚0 disk.sda.out‚ 53.6796875 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚6294.67236328125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚6294.67236328125 disk.sda3.out‚ 53.6796875 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda2.out‚0 cpu‚25.25082299410174 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 53.6796875* proc.memory.percent‚16.558241844177246 network.recv‚567179 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 10.3955078125 disk.sda3.in‚0 network.sent‚ 2.806858e+06 disk.sda2.in‚0$ memory_percent‚99.10051822662354( proc.memory.availableMB‚ 147.37109375 proc.cpu.threads‚17% disk./.usageGB‚291.77343368530273 disk.dm-0.in‚0.078125# proc.memory.rssMB‚ 2712.90234375oß÷2Â- _wandbruntime‚1477  _runtime‚1477pœÇÃÁ _step‚131 & lr-AdamW/pg1‚0.0001534194624781018 & lr-AdamW/pg2‚0.0001534194624781018 trainer/global_step‚1649 % _timestamp‚1.7884417600039198e+09 _runtime‚1477.051145042W`2tÌÄÆ & lr-AdamW/pg1‚0.0001534194624781018 & lr-AdamW/pg2‚0.0001534194624781018 trainer/global_step‚1649 % _timestamp‚1.7884417600039198e+09 _runtime‚1477.051145042 _step‚131ƒ”2Å- _wandbruntime‚1477  _runtime‚1477pDÍ)ë'ÆåO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28951110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127801597612032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64746762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112962454541312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35818760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35795652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17022288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114511509088427 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1549054547115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35818760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112962471580672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28951112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25679343 _step‚132 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1555003860992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25678800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117660469192192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8657055 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35795652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129436702045696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1635323354624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114512307328543 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116105482501632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28951110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117661269487104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117660469192192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 trainer/global_step‚1649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1555003860992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1555786985472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116105465331200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1635129206272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1555786985472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194148352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788441760004609e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328‹îÌ ÇêO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116105465331200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28951110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127801597612032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64746762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112962454541312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35818760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35795652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17022288 XJDeviceStatsMonitor.on’ž¯f$_train_batch_start/requested_bytes.small_pool.current‚ 781200756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114511509088427 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1549054547115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35818760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112962471580672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28951112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25679343 _step‚132 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1555003860992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25678800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117660469192192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8657055 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35795652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129436702045696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1635323354624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114512307328543 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116105482501632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28951110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117661269487104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117660469192192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 trainer/global_step‚1649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1555003860992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1555786985472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116105465331200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1635129206272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1555786985472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194148352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788441760004609e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549978503519 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1555931346432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127877279802368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28968152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35816986 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1636309808128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117730028251136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1556714484224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116174114075136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116174096904704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25694256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127877255029760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113029272330240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64785136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28968150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1556714484224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197936128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25693729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17032277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1636136644608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28968152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129513589610496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64808272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113029289369600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _runtime‚1477.851481267 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117730828559360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35840120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35840120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25693729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17032277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1636136644608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28968152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129513589610496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64808272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113029289369600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _runtime‚1477.851481267 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117730828559360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35840120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35840120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549978503519 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1555931346432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127877279802368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28968152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35816986 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1636309808128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117730028251136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1556714484224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116174114075136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116174096904704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25694256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127877255029760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113029272330240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64785136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28968150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489…bH<2Ë- _wandbruntime‚1477  _runtime‚1477Ýú-¿MÌÇ /loss/FRt2/detection_loss‚2.5346727840828174 1loss/SRqqt2/detection_loss‚2.3711927686968157 .loss/FBt3/detection_loss‚6.727745952350753 _runtime‚1477.852705947 /loss/FRt3/assignment_loss‚6.219935637253982 /loss/FRt4/assignment_loss‚4.406947225790757 0loss/FBt3/assignment_loss‚0.6862117903518603 /loss/FBt4/assignment_loss‚0.585161903156684 0loss/SRqqt4/detection_loss‚0.910339297756793 /loss/FRt1/assignment_loss‚5.842344805046364 /loss/FRt2/assignment_loss‚3.973433868451552 0loss/SRqqt3/detection_loss‚2.605230889938496 .loss/FBt1/detection_loss‚4.130085468292236 -loss/FBt4/detection_loss‚8.38497048157912 1loss/FBt2/assignment_loss‚0.26205965656826075 $ _timestamp‚1.788441760805555e+09 .loss/FRt1/detection_loss‚2.230960373249319 trainer/global_step‚1649 1loss/SRqqt1/assignment_loss‚2.751212273226228 .loss/FRt3/detection_loss‚4.163613649515005 1loss/SRqqt3/assignment_loss‚2.753290856326068 1loss/SRqqt4/assignment_loss‚2.625590628673953 0loss/FBt1/assignment_loss‚0.7443785866101583 /loss/FRt4/detection_loss‚1.9578050457514249 0loss/SRqqt1/detection_loss‚2.371489219887312 %loss/total_loss‚73.06456907982329 epoch‚1 1loss/SRqqt2/assignment_loss‚2.357149863916059 /loss/FBt2/detection_loss‚1.4687460533512586 _step‚134•éŠcRÍÌ /loss/FRt3/assignment_loss‚6.219935637253982 /loss/FRt4/assignment_loss‚4.406947225790757 0loss/FBt3/assignment_loss‚0.6862117903518603 /loss/FBt4/assignment_loss‚0.585161903156684 0loss/SRqqt4/detection_loss‚0.910339297756793 /loss/FRt1/assignment_loss‚5.842344805046364 /loss/FRt2/assignment_loss‚3.973433868451552 0loss/SRqqt3/detection_loss‚2.605230889938496 .loss/FBt1/detection_loss‚4.130085468292236 -loss/FBt4/detection_loss‚8.38497048157912 1loss/FBt2/assignment_loss‚0.26205965656826075 $ _timestamp‚1.788441760805555e+09 .loss/FRt1/detection_loss‚2.230960373249319 trainer/global_step‚1649 1loss/SRqqt1/assignment_loss‚2.751212273226228 .loss/FRt3/detection_loss‚4.163613649515005 1loss/SRqqt3/assignment_loss‚2.753290856326068 1loss/SRqqt4/assignment_loss‚2.625590628673953 0loss/FBt1/assignment_loss‚0.7443785866101583 /loss/FRt4/detection_loss‚1.9578050457514249 0loss/SRqqt1/detection_loss‚2.371489219887312 %loss/total_loss‚73.06456907982329 epoch‚1 1loss/SRqqt2/assignment_loss‚2.357149863916059 /loss/FBt2/detection_loss‚1.4687460533512586 _step‚134 /loss/FRt2/detection_loss‚2.5346727840828174 1loss/SRqqt2/detection_loss‚2.3711927686968157 .loss/FBt3/detection_loss‚6.727745952350753 _runtime‚1477.852705947†§*)͞΂(:“ §éåÔƒ©ÓÇ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍãçKÏ‚(:À §éåÔ’ʺÊ* disk./.usagePercent‚19.581806111854238 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚6339.76220703125# memory_percent‚99.1006851196289$ disk./.usageGB‚291.7830619812012 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.sda1.in‚0 cpu‚25.2488059532757 proc.cpu.threads‚17 disk.sda.out‚ 54.26171875 disk.drbd1716.in‚0" disk.dm-0.out‚6339.76220703125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125% proc.memory.availableMB‚ 147.34375# proc.memory.rssMB‚ 2712.90234375 network.recv‚571036 disk.sda2.out‚0 disk.sda3.out‚ 54.26171875 disk.dm-1.out‚ 54.26171875 network.sent‚ 2.812645e+06 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚6339.76220703125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 10.3955078125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125* proc.memory.percent‚16.558241844177246¥`ajЂ(:’ ¶éåÔ䀾Ç gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.989999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hZ±Ñ‚(:¾ ¶éåÔ¿Ñ¯Ê disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚54.78125 disk.drbd1716.in‚0$ memory_percent‚99.10223484039307* disk./.usagePercent‚19.582778172461747 disk.drbd2265.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6385.41650390625 disk.dm-1.out‚54.78125 network.recv‚577652 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.sda3.out‚54.78125 disk.nvme0n1.in‚0.078125 disk.drbd2265*'¹[2.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚6385.41650390625 disk.dm-0.in‚0.078125" disk.dm-0.out‚6385.41650390625( proc.memory.availableMB‚ 147.08984375# proc.memory.rssMB‚ 2712.90234375 disk.rbd1.out‚ 10.5830078125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.558241844177246 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚0 network.sent‚ 2.921078e+06 cpu‚25.246854718294514% disk./.usageGB‚291.79754638671875 disk.drbd1716.out‚0Ä È(‘Ò‚(:† ÅéåÔ²¦ºÇ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yU¯PÓ‚(:Å ÅéåÔƒë¤É network.recv‚581364 disk.drbd2266.in‚0 disk.rbd1.out‚ 10.5830078125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 55.41796875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 55.41796875 disk.dm-1.in‚0% disk.nvme0n1.out‚6430.79931640625" disk.dm-0.out‚6430.79931640625 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚6430.79931640625 disk.sda3.in‚0' proc.memory.availableMB‚ 147.3046875 disk.sda.out‚ 55.41796875 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 cpu‚25.244975536840283 proc.cpu.threads‚17% disk./.usageGB‚291.80596923828125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚99.10092353820801* proc.memory.percent‚16.558241844177246 network.sent‚ 2.926703e+06) disk./.usagePercent‚19.58334343709735 disk.sda.in‚0# proc.memory.rssMB‚ 2712.90234375-'·W2Ô- _wandbruntime‚1519  _runtime‚1519Nž}ÉÕà _runtime‚1519.691844444 _step‚135 ' lr-AdamW/pg1‚0.00015327063616320193 ' lr-AdamW/pg2‚0.00015327063616320193 trainer/global_step‚1699 % _timestamp‚1.7884418026445887e+090ºM ÎÖÈ _runtime‚1519.691844444 _step‚135 ' lr-AdamW/pg1‚0.00015327063616320193 ' lr-AdamW/pg2‚0.00015327063616320193 trainer/global_step‚1699 % _timestamp‚1.7884418026445887e+09‡YP.2×- _wandbruntime‚1519  _runtime‚1519w›¡ì'ØæO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29803110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26430498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚1699 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1601392546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121138436553216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _step‚136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121139236848128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36865704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117899408186351 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1684306594304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116303361027072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8908613 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66668814 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119537044006400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36888812 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119537044006400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1601392546816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29803110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133270013723136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117898609946235 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29803112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119537061176832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29803112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17521888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1595265958523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602175671296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121139236848128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17521885 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36865704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131585707128832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121138436553216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36888812 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8909132 _runtime‚1519.692746987 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26431020 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1684116640256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756ó@T”ñ'ÙëO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602175671296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121139236848128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17521885 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36865704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131585707128832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121138436553216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36888812 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8909132 _runtime‚1519.692746987 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26431020 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1684116640256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29803110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26430498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚1699 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1601392546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121138436553216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _step‚136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121139236848128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36865704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117899408186351 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1684306594304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116303361027072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8908613 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66668814 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119537044006400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36888812 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119537044006400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1601392546816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29803110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133270013723136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117898609946235 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29803112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119537061176832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29803112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17521888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1595265958523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ˆ« 1²2Ú- _wandbruntime‚1520  _runtime‚1520¬¢}–%ÛïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36887038 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603103170048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131661389319168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 _step‚137 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66730324 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596189914927 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121208795920384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788441803454669e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26445333 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133346669643776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8913971 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117966351691567 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131661364546560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119605692750336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603103170048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29820152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26445851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133346448638976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29820150 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36910172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1685280324608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36910172 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17531880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8913456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66707188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17531877 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66730324 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117967149931891 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_byte•ì†0_s.small_pool.current‚ 196232192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596971115891øn'Öú&ÜôM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602320032256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29820150 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26445333 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133346669643776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8913971 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117966351691567 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131661364546560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119605692750336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603103170048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29820152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26445851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133346448638976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29820150 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36910172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1685280324608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36910172 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17531880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8913456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66707188 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17531877 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66730324 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117967149931891 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596971115891 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36887038 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603103170048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131661389319168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 _step‚137 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66730324 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596189914927 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121208795920384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788441803454669e+09 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135369816645632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122968639852032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1647781232640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192051200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122968622681600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124616403914240 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9159393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119644250473472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68590866 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124617204209152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137102962991616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884418453167279e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37958864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27180881 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9158874 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122968622681600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1648564357120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37935756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135369791873024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30655110 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1641477369931 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30655110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124617204209152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119644233434112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30655112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚1749 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1732954294784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121285710804043 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18021488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1648564357120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124616403914240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121286509044159 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18021485 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137102746167808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37935756 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37958864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1733146345984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122968639852032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27180359 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0½*C”ÄëëO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18021485 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137102746167808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37935756 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37958864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1733146345984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122968639852032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27180359 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135369816645632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122968639852032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1647781232640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192051200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122968622681600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124616403914240 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9159393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119644250473472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68590866 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124617204209152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStÄ馺-atsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137102962991616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884418453167279e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37958864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27180881 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9158874 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122968622681600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1648564357120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37935756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135369791873024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30655110 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1641477369931 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30655110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124617204209152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119644233434112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30655112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚1749 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1732954294784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121285710804043 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18021488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1648564357120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124616403914240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121286509044159 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ŒJq‡2ì- _wandbruntime‚1563  _runtime‚1563š<Êdö&íðM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1642401326335 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68629240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18031477 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137179421188096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124686763281408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68652376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37980224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643182527299 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37957090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 _runtime‚1563.165487686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30672152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123037271425536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68652376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123037254255104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1648708718080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123037254255104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18031480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68629240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1648708718080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135445498835968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121353452549375 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27195326 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30672152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123037271425536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 % _timestamp‚1.7884418461176546e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119711051223040I,ÙÄû&îõM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1648708718080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123037254255104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18031480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68629240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1648708718080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135445498835968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121353452549375 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27195326 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30672152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123037271425536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 % _timestamp‚1.7884418461176546e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119711051223040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1642401326335 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68629240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18031477 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137179421188096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124686763281408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68652376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37980224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643182527299 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37957090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 _runtime‚1563.165487686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30672152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123037271425536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68652376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123037254255104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521>_2ï- _wandbruntime‚1563  _runtime‚1563Gû˜¦WðÑ .loss/FRt2/detection_loss‚2.360943514960153 epoch‚1 % _timestamp‚1.7884418461194599e+09 1loss/SRqqt2/assignment_loss‚2.502628293065798 0loss/FBt1/assignment_loss‚0.6402526479505468 1loss/SRqqt2/detection_loss‚2.5780229965845742 2loss/SRqqt4/assignment_loss‚2.5934597541888555 %loss/total_loss‚64.21044396659522 2loss/SRqqt1/assignment_loss‚2.2201608270406723 0loss/FBt3/assignment_loss‚0.6299287788005188 0loss/FBt4/assignment_loss‚0.7475529052317142 1loss/SRqqt1/detection_loss‚3.1584676081935563 /loss/FBt4/detection_loss‚3.2435360073043285 _step‚142 /loss/FRt3/assignment_loss‚6.081450414657593 2loss/SRqqt3/assignment_loss‚2.2937679244921756 /loss/FRt1/detection_loss‚2.5621655828186443 /loss/FBt2/detection_loss‚0.8935920113984209 .loss/FBt3/detection_loss‚3.189491853117943 /loss/FRt2/assignment_loss‚4.099448112079076 /loss/FRt4/assignment_loss‚4.642166073257859 1loss/SRqqt4/detection_loss‚0.9955697255519529 /loss/FBt1/detection_loss‚4.4604127407073975 0loss/FBt2/assignment_loss‚0.2297940102881352 /loss/FRt3/detection_loss‚3.6264950573444366 1loss/SRqqt3/detection_loss‚2.9190460535196157 trainer/global_step‚1749 _runtime‚1563.166582993 0loss/FRt1/assignment_loss‚5.6572018294107345 -loss/FRt4/detection_loss‚1.88488924463053Ú\ñÖ /loss/FRt2/assignment_loss‚4.099448112079076 /loss/FRt4/assignment_loss‚4.642166073257859 1loss/SRqqt4/detection_loss‚0.9955697255519529 /loss/FBt1/detection_loss‚4.4604127407073975 0loss/FBt2/assignment_loss‚0.2297940102881352 /loss/FRt3/detection_loss‚3.6264950573444366 1loss/SRqqt3/detection_loss‚2.9190460535196157 trainer/global_step‚1749 _runtime‚1563.166582993 0loss/FRt1/assignment_loss‚5.6572018294107345 -loss/FRt4/detection_loss‚1.88488924463053 .loss/FRt2/detection_loss‚2.360943514960153 epoch‚1 % _timestamp‚1.7884418461194599e+09 1loss/SRqqt2/assignment_loss‚2.502628293065798 0loss/FBt1/assignment_loss‚0.6402526479505468 1loss/SRqqt2/detection_loss‚2.5780229965845742 2loss/SRqqt4/assignment_loss‚2.5934597541888555 %loss/total_loss‚64.21044396659522 2loss/SRqqt1/assignment_loss‚2.2201608270406723 0loss/FBt3/assignment_loss‚0.6299287788005188 0loss/FBt4/assignment_loss‚0.7475529052317142 1loss/SRqqt1/detection_loss‚3.1584676081935563 /loss/FBt4/detection_loss‚3.2435360073043285 _step‚142 /loss/FRt3/assignment_loss‚6.081450414657593 2loss/SRqqt3/assignment_loss‚2.2937679244921756 /loss/FRt1/detection_loss‚2.5621655828186443 /loss/FBt2/detection_loss‚0.8935920113984209 .loss/FBt3/detection_loss‚3.189491853117943ŽÁÅ*žò‚(:“ êåÔ¾Î·Ç gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.294199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XVû@ó‚(:µ êåÔÆјÊ disk.dm-1.out‚ 57.58984375 disk.drbd2080.in‚0 disk.sda1.out‚0$ memory_percent‚99.10366535186768 proc.memory.rssMB‚ 2712.90625 cpu‚25.237483410668727 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 network.recv‚599901 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚291.8734588623047# disk.nvme0c0n1.out‚ 6622.8203125 disk.sda.out‚ 57.58984375! disk.nvme0n1.out‚ 6622.8203125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚19.58787272239313 disk.sda.in‚0 disk.sda3.out‚ 57.58984375 disk.dm-1.in‚0 disk.dm-0.out‚ 6622.8203125 disk.sda2.out‚0 disk.rbd1.out‚ 10.888671875 network.sent‚ 3.052606e+06 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 146.85546875 disk.nvme0c0n1.in‚0.078125* proc.memory.percent‚16.558265686035156¦Ÿ©F“ô‚(:ˆ êåÔúÀˆÇ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñl~Iõ‚(:¾ êåÔ–äšÊ network.recv‚605768 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚6672.76025390625 disk.drbd2080.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚6672.76025390625% disk./.usageGB‚291.88570404052734 disk.sda1.in‚0 disk.dm-1.out‚ 58.0859375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0# memory_percent‚99.1037130355835 cpu‚25.235897444480795 disk.dm-0.in‚0.078125) disk./.usagePercent‚19.58869450657804 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6672.76025390625 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125( proc.memory.availableMB‚ 146.84765625* proc.memory.percent‚16.558265686035156 network.sent‚ 3.160675e+06 disk.sda2.in‚0 disk.drbd2266.in‚0 proc.memory.rssMB‚ 2712.90625 proc.cpu.threads‚17 disk.rbd1.out‚ 11.130859375 disk.sda.in‚0 disk.sda3.out‚ 58.0859375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 58.0859375 disk.sda2.out‚0 disk.drbd1716.in‚0'g 2ö- _wandbruntime‚1604  _runtime‚1604_‘^pÇ÷Á ' lr-AdamW/pg2‚0.00015292534634509701 trainer/global_step‚1799 # _timestamp‚1.78844188777029e+09 _runtime‚1604.817608819 _step‚143 ' lr-AdamW/pg1‚0.00015292534634509701 ±ÔÌøÆ _step‚143 ' lr-AdamW/pg1‚0.00015292534634509701 ' lr-AdamW/pg2‚0.00015292534634509701 trainer/global_step‚1799 # _timestamp‚1.78844188777029e+09 _runtime‚1604.817608819¶ê¶2ù- _wandbruntime‚1604  _runtime‚1604%%ªúæO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139153926162432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139153901389824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStaÛ'ÊB&tsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128094371275264 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70512918 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140936004804608 _runtime‚1604.818587696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128094371275264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31507110 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140935792175104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1694169918464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9409937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39005808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39028916 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39005808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187856896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126400218527232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1781890785280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1694169918464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31507110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126400201356800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128095171570176 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124673609901967 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124672811661851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31507112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18521088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126400218527232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128095171570176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124673609901967 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124672811661851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31507112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18521088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126400218527232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128095171570176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128094371275264 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70512918 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140936004804608 _runtime‚1604.818587696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128094371275264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31507110 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140935792175104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1694169918464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9409937 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39005808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39028916 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010—MKX“ü‚(:ˆ ŸêåÔÄóÝÈ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜùUˆFý‚(:» ŸêåÔŸâèÉ disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2712.90625 cpu‚25.23419988495871 proc.cpu.threads‚17 network.recv‚610011% disk./.usageGB‚291.90808486938477 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 11.130859375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6718.97119140625 disk.sda2.in‚0 disk.dm-1.out‚ 58.4765625 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125$ proc.memory.availableMB‚146.5625* proc.memory.percent‚16.558265686035156% disk.nvme0n1.out‚6718.97119140625 disk.sda1.out‚0$ memory_percent‚99.10545349121094 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 58.4765625 disk.sda2.out‚0* disk./.usagePercent‚19.590196502781428 disk.drbd2265.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚6718.97119140625 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 58.4765625 network.sent‚ 3.166521e+06 disk.drbd2266.in‚0 disk.rbd0.in‚0é|à¾2þ- _wandbruntime‚1605  _runtime‚1605¢iCö&ÿðM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124740553407183 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689393938707 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31524152 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70574428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70574428 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70551292 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128163930334208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚145 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123051940669440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128164730642432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141012671144448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1799 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9414861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139229608352768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1695097403904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128164730642432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9415371 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139229583580160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884418885721214e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126468832930304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1695880541696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1695880541696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27945938 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31524152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18531077 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126468850100736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123051957708800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126468832930304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31524150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1783062791680cÃV²›€õM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚145 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requesteîé8`"d_bytes.large_pool.freed‚123051940669440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128164730642432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141012671144448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1799 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9414861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139229608352768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1695097403904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128164730642432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9415371 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139229583580160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884418885721214e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126468832930304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1695880541696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1695880541696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27945938 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31524152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18531077 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126468850100736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123051957708800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126468832930304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31524150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1783062791680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124740553407183 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689393938707 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31524152 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70574428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70574428 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70551292 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128163930334208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‘!%E2- _wandbruntime‚1605  _runtime‚1605áŠU‚Ï trainer/global_step‚1799 /loss/FRt1/assignment_loss‚5.854662884836611 0loss/FBt2/assignment_loss‚0.2007564138314339 /loss/FRt3/detection_loss‚2.8660697724137987 _runtime‚1605.620842505 /loss/FBt4/detection_loss‚2.4984117929752054 /loss/FRt2/assignment_loss‚3.882871653171296 1loss/SRqqt1/assignment_loss‚2.905733701160976 2loss/SRqqt4/assignment_loss‚2.4495975170779016 .loss/FBt1/detection_loss‚6.543762803077698 1loss/SRqqt2/assignment_loss‚1.495020762485053 /loss/FBt3/detection_loss‚3.9602242410182953 1loss/SRqqt3/detection_loss‚2.7287775073732647 1loss/SRqqt4/detection_loss‚1.0493416075421647 /loss/FBt2/detection_loss‚0.9068166121819846 %loss/total_loss‚64.54259590785341 /loss/FRt1/detection_loss‚2.4233067456794823 .loss/FRt2/detection_loss‚2.086745875947019 /loss/FRt3/assignment_loss‚5.606141899313245 /loss/FRt4/assignment_loss‚4.310747295009847 1loss/SRqqt3/assignment_loss‚2.984855389595032 0loss/FBt3/assignment_loss‚0.8438833342341241 0loss/FBt4/assignment_loss‚0.5421905334200207 0loss/SRqqt2/detection_loss‚2.838663358773504 1loss/SRqqt1/detection_loss‚2.5625305601528714 0loss/FBt1/assignment_loss‚0.5690043568611145 /loss/FRt4/detection_loss‚2.4324792897214693 epoch‚1 % _timestamp‚1.7884418885737357e+09 _step‚146ÄIu¨ZƒÔ 1loss/SRqqt2/assignment_loss‚1.495020762485053 /loss/FBt3/detection_loss‚3.9602242410182953 1loss/SRqqt3/detection_loss‚2.7287775073732647 1loss/SRqqt4/detection_loss‚1.0493416075421647 /loss/FBt2/detection_loss‚0.9068166121819846 %loss/total_loss‚64.54259590785341 /loss/FRt1/detection_loss‚2.4233067456794823 .loss/FRt2/detection_loss‚2.086745875947019 /loss/FRt3/assignment_loss‚5.606141899313245 /loss/FRt4/assignment_loss‚4.310747295009847 1loss/SRqqt3/assignment_loss‚2.984855389595032 0loss/FBt3/assignment_loss‚0.8438833342341241 0loss/FBt4/assignment_loss‚0.5421905334200207 0loss/SRqqt2/detection_loss‚2.838663358773504 1loss/SRqqt1/detection_loss‚2.5625305601528714 0loss/FBt1/assignment_loss‚0.5690043568611145 /loss/FRt4/detection_loss‚2.4324792897214693 epoch‚1 % _timestamp‚1.7884418885737357e+09 _step‚146 trainer/global_step‚1799 /loss/FRt1/assignment_loss‚5.854662884836611 0loss/FBt2/assignment_loss‚0.2007564138314339 /loss/FRt3/detection_loss‚2.8660697724137987 _runtime‚1605.620842505 /loss/FBt4/detection_loss‚2.4984117929752054 /loss/FRt2/assignment_loss‚3.882871653171296 1loss/SRqqt1/assignment_loss‚2.905733701160976 2loss/SRqqt4/assignment_loss‚2.4495975170779016 .loss/FBt1/detection_loss‚6.543762803077698’{üÊ’ž„‚(:“ ®êåÔéê­Ç gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚137.582' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.516399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M a>…‚(:³ ®êåÔ­Ú»É disk.drbd2080.out‚0 disk.sda3.out‚ 59.234375 disk.dm-1.in‚0 disk.rbd1.out‚ 11.490234375) disk./.usagePercent‚19.59217800109645 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.dm-0.out‚6796.740234375 disk.drbd2265.out‚0* proc.memory.percent‚16.558265686035156$ disk./.usageGB‚291.9376106262207' proc.memory.availableMB‚ 146.5234375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2712.90625 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 59.234375# disk.nvme0n1.out‚6796.740234375 disk.drbd2266.in‚0 cpu‚25.23253275256723 network.sent‚ 3.263669e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda.out‚ 59.234375 disk.drbd2080.in‚0$ memory_percent‚99.10569190979004 proc.cpu.threads‚17 network.recv‚616590% disk.nvme0c0n1.out‚6796.740234375 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125SÚ–µ‘†‚(:† ½êåÔ‰ŠÉÇ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÎjC‡‚(:¸ ½êåÔ¤§ÊÊ% disk.nvme0c0n1.out‚6843.427734375' proc.memory.availableMB‚ 146.7734375$ disk./.usageGB‚291.9489288330078 disk.sda3.in‚0# disk.nvme0n1.out‚6843.427734375 disk.rbd1.in‚0 disk.sda1.in‚0 network.sent‚ 3.360547e+06 disk.sda3.out‚ 59.8984375 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 11.490234375 disk.sda.in‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.558265686035156 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.dm-1.in‚0 disk.dm-1.out‚ 59.8984375 disk.drbd2266.out‚0 proc.memory.rssMB‚ 2712.90625 disk.dm-0.out‚6843.427734375 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda.out‚ 59.8984375$ memory_percent‚99.10416603088379 cpu‚25.230849588217524 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚621175* disk./.usagePercent‚19.592937575450538 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.rbd0.out‚03±ë92ˆ- _wandbruntime‚1647  _runtime‚1647u<å‰É‰Ã ' lr-AdamW/pg1‚0.00015272895479977254 ' lr-AdamW/pg2‚0.00015272895479977254 trainer/global_step‚1849 % _timestamp‚1.7884419301349955e+09 _runtime‚1647.182240148 _step‚147œ¹ÎŠÈ % _timestamp‚1.7884419301349955e+09 _runtime‚1647.182240148 _step‚147 ' lr-AdamW/pg1‚0.00015272895479977254 ' lr-AdamW/pg2‚0.00015272895479977254 trainer/global_step‚1849“GÆrs2‹- _wandbruntime‚1647  _runtime‚1647°±‡ì'ŒæO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72458080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142938035679232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1740558604288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1740558604288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734681393503 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32359112 _runtime‚1647.183239103 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72434970 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28681751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9661063 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1741341728768 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32359110 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126326029366272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131573138931200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1733900192747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19020685 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144768986239488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 _step‚148 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131572338636288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40098968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28681216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19020688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40075860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129831780032000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129831797202432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212629504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131572338636288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1830950560256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187856896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884419301357946e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32359112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32359110èpšÆëO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32359110 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72458080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142938035679232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1740558604288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1740558604288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734681393503 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32359112 _runtime‚1647.183239103 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72434970 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28681751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9661063 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1741341728768 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32359110 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126326029366272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131573138931200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1733900192747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19020685 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144768986239488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 _step‚148 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131572338636288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40098968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28681216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19020688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40075860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129831780032000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129831797202432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212629504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131572338636288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1830950560256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187856896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884419301357946e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32359112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912”Û_Ù¬2Ž- _wandbruntime‚1647  _runtime‚1647ðîŒõ&ïM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144845651978240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128127654264991 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32376150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40120328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126392830115840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144845445653504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9666121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181552128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32376152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129900411605504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9665593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28696270 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1741486089728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19030677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128128452505315 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734824149151 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 trainer/global_step‚1849 $ _timestamp‚1.788441930930264e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28696801 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126392847155200 _runtime‚1647.977992035 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1742269227520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72496480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32376150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131641897695232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40097194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206324736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40097194 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129900411605504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32376152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72473344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1831934108672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72473344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40120328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 17356053501159è]ú&ôM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128128452505315 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734824149151 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 trainer/global_step‚1849 $ _timestamp‚1.788441930930264e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28696801 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126392847155200 _runtime‚1647.977992035 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1742269227520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72496480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32376150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131641897695232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40097194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206324736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40097194 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129900411605504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32376152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72473344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1831934108672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72473344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40120328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1735605350115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144845651978240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128127654264991 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32376150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40120328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126392830115840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144845445653504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9666121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181552128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32376152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129900411605504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9665593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28696270 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1741486089728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19030677•C/ë2‘- _wandbruntime‚1647  _runtime‚1647ˆo Q’Ë 1loss/SRqqt2/assignment_loss‚2.454659279216738 0loss/FBt4/assignment_loss‚0.4069656316093063 1loss/SRqqt2/detection_loss‚2.9450954163775727 .loss/FBt4/detection_loss‚5.956923243674365 /loss/FRt2/assignment_loss‚4.676850725303996 0loss/SRqqt1/detection_loss‚2.526690018527648 1loss/SRqqt4/detection_loss‚1.0014090595841407 2loss/SRqqt4/assignment_loss‚2.2428341097831725 1loss/FBt2/assignment_loss‚0.28475512246384826 /loss/FBt2/detection_loss‚0.8328562787159657 .loss/FRt3/detection_loss‚3.050143389701843 /loss/FBt3/detection_loss‚3.1440470048359463 epoch‚1 .loss/FRt2/detection_loss‚2.767710928664063 /loss/FRt4/assignment_loss‚4.076691506338901 0loss/FBt1/assignment_loss‚0.7710686127344767 /loss/FRt4/detection_loss‚2.0001238921626667 %loss/total_loss‚65.73343746207851 $ _timestamp‚1.788441930932041e+09 2loss/SRqqt1/assignment_loss‚2.5159588403561535 0loss/SRqqt3/detection_loss‚2.644250301378114 .loss/FRt1/detection_loss‚2.270328286588192 /loss/FRt3/assignment_loss‚5.371780548095703 1loss/SRqqt3/assignment_loss‚3.048784541232245 .loss/FBt1/detection_loss‚4.539466619491577 _runtime‚1647.979153257 _step‚150 /loss/FRt1/assignment_loss‚5.574486827850341 /loss/FBt3/assignment_loss‚0.629557277391541 trainer/global_step‚1849Ð3ºÔV“Ð 0loss/SRqqt1/detection_loss‚2.526690018527648 1loss/SRqqt4/detection_loss‚1.0014090595841407 2loss/SRqqt4/assignment_loss‚2.2428341097831725 1loss/FBt2/assignment_loss‚0.28475512246384826 /loss/FBt2/detection_loss‚0.8328562787159657 .loss/FRt3/detection_loss‚3.050143389701843 /loss/FBt3/detection_loss‚3.1440470048359463 epoch‚1 .loss/FRt2/detection_loss‚2.767710928664063 /loss/FRt4/assignment_loss‚4.076691506338901 0loss/FBt1/assignment_loss‚0.7710686127344767 /loss/FRt4/detection_loss‚2.0001238921626667 %loss/total_loss‚65.73343746207851 $ _timestamp‚1.788441930932041e+09 2loss/SRqqt1/assignment_loss‚2.5159588403561535 0loss/SRqqt3/detection_loss‚2.644250301378114 .loss/FRt1/detection_loss‚2.270328286588192 /loss/FRt3/assignment_loss‚5.371780548095703 1loss/SRqqt3/assignment_loss‚3.048784541232245 .loss/FBt1/detection_loss‚4.539466619491577 _runtime‚1647.979153257 _step‚150 /loss/FRt1/assignment_loss‚5.574486827850341 /loss/FBt3/assignment_loss‚0.629557277391541 trainer/global_step‚1849 1loss/SRqqt2/assignment_loss‚2.454659279216738 0loss/FBt4/assignment_loss‚0.4069656316093063 1loss/SRqqt2/detection_loss‚2.9450954163775727 .loss/FBt4/detection_loss‚5.956923243674365 /loss/FRt2/assignment_loss‚4.676850725303996–ÓåÇ“”‚(:ˆ ÌêåÔ­Ã³Ç gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à­0ÕF•‚(:» ÌêåÔùͪÊ disk.sda.out‚ 60.37890625 network.sent‚ 3.366631e+06% disk./.usageGB‚291.96056747436523% disk.nvme0c0n1.out‚6906.341796875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.drbd1716.out‚0 proc.memory.rssMB‚ 2712.90625 network.recv‚625748) disk./.usagePercent‚19.59371865443751 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 11.5224609375& proc.memory.availableMB‚ 146.734375* proc.memory.percent‚16.558265686035156 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.sda3.out‚ 60.37890625 disk.dm-0.out‚6906.341796875 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚6906.341796875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 cpu‚25.229089633231144 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-1.out‚ 60.37890625$ memory_percent‚99.10440444946289 proc.cpu.threads‚17D­ÿž–‚(:“ ÛêåÔõ˜¬Ç gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.974200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úy}É?—‚(:´ ÛêåÔºŒ²Ê# disk.nvme0n1.out‚6947.576171875 disk.drbd2266.out‚0$ memory_percent‚99.10597801208496' proc.memory.availableMB‚ 146.4765625 disk.rbd1.out‚ 11.6162109375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.recv‚631792 disk.nvme0n1.in‚0.078125 disk.sda3.in‚0* proc.memory.percent‚16.558265686035156) disk./.usagePercent‚19.59452354212516 disk.dm-1.out‚ 60.984375 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚6947.576171875 disk.sda2.out‚0% disk./.usageGB‚291.97256088256836 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.rbd0.in‚0 network.sent‚ 3.475076e+06 disk.dm-0.out‚6947.576171875 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.sda.out‚ 60.984375 proc.memory.rssMB‚ 2712.90625 cpu‚25.2274992273383 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 60.984375Zž˜‚(:“ êêåÔÜ«¿Ç gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.017' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.203399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v&@åC™‚(:¸ êêåÔ¾€ÙÊ) disk./.usagePercent‚19.59511722359706 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.drbd2266.in‚0 network.sent‚ 3.48126e+06% disk./.usageGB‚291.98140716552734 disk.sda1.out‚0 disk.sda.out‚61.59375 disk.dm-1.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚6990.21337890625 disk.drbd2265.in‚0 disk.sda1.in‚0$ memory_percent‚99.10445213317871 proc.memory.rssMB‚ 2712.90625 disk.sda3.in‚0 disk.dm-1.out‚61.59375 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚61.59375 cpu‚25.225782677471656 network.recv‚636223 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2266.out‚0 disk.rbd1.out‚ 11.8896484375% disk.nvme0n1.out‚6990.21337890625' proc.memory.availableMB‚ 146.7265625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6990.21337890625* proc.memory.percent‚16.558265686035156ˆk2š- _wandbruntime‚1689  _runtime‚1689,Á ɛà ' lr-AdamW/pg1‚0.00015251678563501782 ' lr-AdamW/pg2‚0.00015251678563501782 trainer/global_step‚1899 % _timestamp‚1.7884419723894749e+09 _runtime‚1689.436674577 _step‚151ð6ÆÎœÈ ' lr-AdamW/pg2‚0.00015251678563501782 trainer/global_step‚1899 % _timestamp‚1.7884419723894749e+09 _runtime‚1689.436674577 _step‚151 ' lr-AdamW/pg1‚0.00015251678563501782—sÐ2- _wandbruntime‚1689  _runtime‚1689[©}œËžåO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9912110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33211110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41145912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1786947290112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19520288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133263358707200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780892804911 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1780111604155 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _step‚152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135051106292224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133263375877632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1787730414592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19520285 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33211112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41169020 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148601961839104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210532352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33211110 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148601751306752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131447013377467 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74357022 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41169020 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788441972390232e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135051106292224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135050305997312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133263375877632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1786947290112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41145912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759744 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74380132 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29431871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129666918812672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1787730414592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146722120423424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131447811617583 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129666901773312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _step‚152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135051106292224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133263375877632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1787730414592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19520285 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33211112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41169020 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148601961839104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210532352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33211110 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148601751306752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131447013377467 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74357022 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41169020 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788441972390232e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135051106292224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135050305997312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133263375877632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1786947290112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41145912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759744 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74380132 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29431871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129666918812672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1787730414592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146722120423424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131447811617583 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129666901773312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29446951 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133332007451136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133331990280704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1788657913344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148678623290368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33228150 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33228150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19530277 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135119865056256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189940736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133332007451136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚1899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41167246 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74418532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74395396 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33228152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41190380 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41167246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9916674 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880795904000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884419731963003e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1690.244111138 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781816761523 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29447483 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131515553363123 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 _step‚153 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1787874775552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2x]ॢõM % _timestamp‚1.7884419731963003e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1690.244111138 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781816761523 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29447483 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131515553363123 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 _step‚153 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1787874775552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19530280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146797802613760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1781035560559 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74418532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1787874775552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29446951 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133332007451136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133331990280704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1788657913344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148678623290368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ÆxgÍVP@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33228150 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33228150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19530277 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135119865056256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189940736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133332007451136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚1899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41167246 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74418532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74395396 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33228152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41190380 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41167246 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9916674 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880795904000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0™ÌÖÄZ2£- _wandbruntime‚1690  _runtime‚1690!i'W¤Ñ /loss/FRt1/detection_loss‚2.2040284172232663 1loss/SRqqt3/detection_loss‚2.9910599926243657 trainer/global_step‚1899 % _timestamp‚1.7884419731980255e+09 /loss/FRt3/assignment_loss‚5.418053195590065 2loss/SRqqt4/assignment_loss‚2.3310595793383464 .loss/FRt2/detection_loss‚2.038121920256388 0loss/FBt1/assignment_loss‚0.6649103959401449 0loss/FBt2/assignment_loss‚0.2206363421609666 %loss/total_loss‚60.68970748669358 0loss/FRt2/assignment_loss‚3.7056858979520344 1loss/SRqqt3/assignment_loss‚2.711971218171327 /loss/FRt4/detection_loss‚1.9134221535826486 .loss/FBt1/detection_loss‚3.868366241455078 0loss/FRt4/assignment_loss‚3.9829505322471497 /loss/FBt3/detection_loss‚2.4518435709178448 _step‚154 1loss/SRqqt1/assignment_loss‚2.580638507174121 2loss/SRqqt2/assignment_loss‚2.4313292350423965 1loss/SRqqt1/detection_loss‚2.5502114817500114 0loss/SRqqt2/detection_loss‚2.697518339282588 /loss/FBt4/detection_loss‚3.8659576845322072 epoch‚1 _runtime‚1690.245126093 /loss/FRt1/assignment_loss‚5.684343882969448 0loss/FBt4/assignment_loss‚0.4475786004743116 /loss/FRt3/detection_loss‚3.3675242492130826 /loss/FBt2/detection_loss‚0.7243072904762812 /loss/FBt3/assignment_loss‚0.746508440002799 1loss/SRqqt4/detection_loss‚1.0916803183167108ÁÀõo\¥Ö /loss/FBt3/assignment_loss‚0.746508440002799 1loss/SRqqt4/detection_loss‚1.0916803183167108 /loss/FRt1/detection_loss‚2.2040284172232663 1loss/SRqqt3/detection_loss‚2.9910599926243657 trainer/global_step‚1899 % _timestamp‚1.7884419731980255e+09 /loss/FRt3/assignment_loss‚5.418053195590065 2loss/SRqqt4/assignment_loss‚2.3310595793383464 .loss/FRt2/detection_loss‚2.038121920256388 0loss/FBt1/assignment_loss‚0.6649103959401449 0loss/FBt2/assignment_loss‚0.2206363421609666 %loss/total_loss‚60.68970748669358 0loss/FRt2/assignment_loss‚3.7056858979520344 1loss/SRqqt3/assignment_loss‚2.711971218171327 /loss/FRt4/detection_loss‚1.9134221535826486 .loss/FBt1/detection_loss‚3.868366241455078 0loss/FRt4/assignment_loss‚3.9829505322471497 /loss/FBt3/detection_loss‚2.4518435709178448 _step‚154 1loss/SRqqt1/assignment_loss‚2.580638507174121 2loss/SRqqt2/assignment_loss‚2.4313292350423965 1loss/SRqqt1/detection_loss‚2.5502114817500114 0loss/SRqqt2/detection_loss‚2.697518339282588 /loss/FBt4/detection_loss‚3.8659576845322072 epoch‚1 _runtime‚1690.245126093 /loss/FRt1/assignment_loss‚5.684343882969448 0loss/FBt4/assignment_loss‚0.4475786004743116 /loss/FRt3/detection_loss‚3.3675242492130826 /loss/FBt2/detection_loss‚0.7243072904762812šxè-¿“¦‚(:ˆ ùêåÔ®¡´Ç gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.946' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É&½}K§‚(:À ùêåÔ¾©ÉË network.recv‚639652 disk.sda3.in‚0' proc.memory.availableMB‚ 146.6640625 disk.dm-1.in‚0 disk.dm-0.in‚0.078125' disk.nvme0c0n1.out‚7032.74072265625$ memory_percent‚99.10483360290527 disk.dm-1.out‚ 62.09765625 disk.rbd1.out‚ 11.8896484375 disk.nvme0c0n1.in‚0.078125 cpu‚25.22422399424586) disk./.usagePercent‚19.59560312589705 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚16.558265686035156 proc.cpu.threads‚17 network.sent‚ 3.486701e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ disk./.usageGB‚291.9886474609375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚7032.74072265625 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda.out‚ 62.09765625 disk.sda3.out‚ 62.09765625 proc.memory.rssMB‚ 2712.90625 disk.drbd1716.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚7032.74072265625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0óùý¨‚(:’ ˆëåÔžÊŸÇ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.33' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.666000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þµn$J©‚(:¿ ˆëåÔÄð·Ê! disk.dm-0.out‚7079.5888671875 disk.rbd1.out‚ 11.9521484375 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0' proc.memory.availableMB‚ 146.8984375 network.recv‚646269 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 62.80859375$ memory_percent‚99.10340309143066 proc.memory.rssMB‚ 2712.90625* disk./.usagePercent‚19.596366540364155 disk.rbd1.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚7079.5888671875 disk.sda3.out‚ 62.80859375$ disk.nvme0n1.out‚7079.5888671875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 3.594901e+06$ disk./.usageGB‚292.0000228881836 disk.drbd2265.out‚0 disk.sda.out‚ 62.80859375 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚0* proc.memory.percent‚16.558265686035156 cpu‚25.222609009889997 õ£“ª‚(:ˆ —ëåÔ¹ö Ç gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚192.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H~ÕH«‚(:½ —ëåÔ¾Ã§É network.recv‚650344 disk.drbd1716.in‚0$ memory_percent‚99.10340309143066 cpu‚25.221021274190175 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.9521484375$ disk.nvme0n1.out‚7121.2607421875 disk.drbd2266.in‚0 disk.sda.out‚ 63.35546875' proc.memory.availableMB‚ 146.8984375$ disk./.usageGB‚292.0035171508789 disk.sda1.out‚0 proc.memory.rssMB‚ 2712.90625 disk.drbd2080.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚7121.2607421875 disk.sda.in‚0) disk./.usagePercent‚19.59660104326552 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 63.35546875 disk.dm-1.out‚ 63.35546875! disk.dm-0.out‚7121.2607421875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 3.60068e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125* proc.memory.percent‚16.558265686035156 proc.cpu.threads‚17ŸÞ2¬- _wandbruntime‚1731  _runtime‚1731Ûu1É­Ã ' lr-AdamW/pg1‚0.00015228888306656875 ' lr-AdamW/pg2‚0.00015228888306656875 trainer/global_step‚1949 % _timestamp‚1.7884420148069077e+09 _runtime‚1731.854102372 _step‚155‹ýVMÎ®È ' lr-AdamW/pg1‚0.00015228888306656875 ' lr-AdamW/pg2‚0.00015228888306656875 trainer/global_step‚1949 % _timestamp‚1.7884420148069077e+09 _runtime‚1731.854102372 _step‚155›û胔2¯- _wandbruntime‚1731  _runtime‚1731¦­}Eî'°èO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136694937382400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10161677 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138529073653248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34063110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150506229940224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138528273358336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1834119100416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1833335975936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1834119100416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76279074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42215964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1928822438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1928638775808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827104216319 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚1949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884420148075633e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1826323015563 _runtime‚1731.854997441 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76302184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42215964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30181562 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136694954552832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42239072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42215964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30181562 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136694954552832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42239072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138528273358336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1834119100416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208435200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1833335975936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1834119100416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76279074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 L?DeviceStatsMonitor.on_train_batch_startš|ñ™'/active.small_pool.freed‚42215964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1928822438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1928638775808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827104216319 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚1949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0œtÜÐD2²- _wandbruntime‚1732  _runtime‚1732Š£ïƒ÷&³ñM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138598632725504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1835046599168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136763568955904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76317448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20029880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138598632725504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10167070 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133074609008640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34080150 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152511743344640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134902654220931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136763586126336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1834263461376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _step‚157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76340584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133074626048000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚1949 _runtime‚1732.660868145 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136763586126336 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42237298 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30196442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828028172931 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788442015613053e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76317448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150581936903168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183649280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138597832417280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10166565 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134901855980607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34080150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1827246971967 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1929622792192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ùÈËü&´öM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1835046599168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136763568955904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76317448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20029880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138598632725504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10167070 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133074609008640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34080150 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152511743344640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134902654220931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136763586126336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1834263461376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _step‚157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76340584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133074626048000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚1949 _runtime‚1732.660868145 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136763586126336 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42237298 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30196442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828028172931 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788442015613053e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76317448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150581936903168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183649280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138597832417280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10166565 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134901855980607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34080150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1827246971967 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1929622792192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138598632725504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328º^2µ- _wandbruntime‚1732  _runtime‚1732ot—U¶Ï /loss/FRt1/detection_loss‚2.5632814753123303 epoch‚1 _step‚158 1loss/SRqqt1/assignment_loss‚2.328186547383666 1loss/SRqqt3/detection_loss‚2.5648116451852463 .loss/FBt2/detection_loss‚1.083227940238026 /loss/FBt3/detection_loss‚3.9091167002916336 0loss/FRt4/assignment_loss‚3.9670523699015785 1loss/SRqqt2/assignment_loss‚2.548866139894182 0loss/SRqqt2/detection_loss‚2.542926875027743 0loss/FBt1/assignment_loss‚0.5323936939239502 0loss/FBt3/assignment_loss‚0.6004770685176481 0loss/SRqqt1/detection_loss‚2.555875202640891 /loss/FRt1/assignment_loss‚5.289058582349257 /loss/FRt3/assignment_loss‚4.850175352657542 2loss/SRqqt4/assignment_loss‚2.3507067792945437 1loss/FBt2/assignment_loss‚0.26173545839436785 /loss/FRt3/detection_loss‚2.8095723530825447 /loss/FBt4/detection_loss‚2.2300995967604895 trainer/global_step‚1949 _runtime‚1732.661912707 .loss/FRt2/detection_loss‚2.038050688164575 /loss/FRt4/detection_loss‚1.7912046315327084 0loss/FRt2/assignment_loss‚3.6450812544141495 1loss/SRqqt3/assignment_loss‚2.822462365907781 0loss/FBt4/assignment_loss‚0.6314909673624656 1loss/SRqqt4/detection_loss‚0.9276085495653134 -loss/FBt1/detection_loss‚6.90743362903595 &loss/total_loss‚61.750895866838576 $ _timestamp‚1.788442015614792e+09V%¶›Z·Ô /loss/FBt4/detection_loss‚2.2300995967604895 trainer/global_step‚1949 _runtime‚1732.661912707 .loss/FRt2/detection_loss‚2.038050688164575 /loss/FRt4/detection_loss‚1.7912046315327084 0loss/FRt2/assignment_loss‚3.6450812544141495 1loss/SRqqt3/assignment_loss‚2.822462365907781 0loss/FBt4/assignment_loss‚0.6314909673624656 1loss/SRqqt4/detection_loss‚0.9276085495653134 -loss/FBt1/detection_loss‚6.90743362903595 &loss/total_loss‚61.750895866838576 $ _timestamp‚1.788442015614792e+09 /loss/FRt1/detection_loss‚2.5632814753123303 epoch‚1 _step‚158 1loss/SRqqt1/assignment_loss‚2.328186547383666 1loss/SRqqt3/detection_loss‚2.5648116451852463 .loss/FBt2/detection_loss‚1.083227940238026 /loss/FBt3/detection_loss‚3.9091167002916336 0loss/FRt4/assignment_loss‚3.9670523699015785 1loss/SRqqt2/assignment_loss‚2.548866139894182 0loss/SRqqt2/detection_loss‚2.542926875027743 0loss/FBt1/assignment_loss‚0.5323936939239502 0loss/FBt3/assignment_loss‚0.6004770685176481 0loss/SRqqt1/detection_loss‚2.555875202640891 /loss/FRt1/assignment_loss‚5.289058582349257 /loss/FRt3/assignment_loss‚4.850175352657542 2loss/SRqqt4/assignment_loss‚2.3507067792945437 1loss/FBt2/assignment_loss‚0.26173545839436785 /loss/FRt3/detection_loss‚2.8095723530825447žY¿”쓸‚(:ˆ ¦ëåÔØÖºÇ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%åüëE¹‚(:º ¦ëåÔЈ¸É disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 63.7890625 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.558265686035156" disk./.usageGB‚292.02880859375" disk.dm-0.out‚7154.47021484375 cpu‚25.21956475620317 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7154.47021484375 disk.dm-1.out‚ 63.7890625 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.sda3.out‚ 63.7890625 disk.drbd2265.in‚0$ memory_percent‚99.10669326782227 network.sent‚ 3.607086e+06 network.recv‚655070 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 12.1396484375 disk.sda1.out‚0& proc.memory.availableMB‚ 146.359375) disk./.usagePercent‚19.59829837321753 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚7154.47021484375 proc.memory.rssMB‚ 2712.90625 proc.cpu.threads‚17 disk.rbd0.in‚0Ûê¾ë“º‚(:ˆ µëåÔ¦ßžÉ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<êH»‚(:½ µëåÔ‰ÃšË network.sent‚ 3.714369e+06 network.recv‚660069 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0$ disk./.usageGB‚292.0335578918457 disk.drbd1716.out‚0 disk.sda.out‚ 64.4140625 disk.dm-1.in‚0 disk.drbd1716.in‚0" disk.dm-0.out‚7201.65771484375 disk.rbd1.out‚ 12.2255859375 proc.memory.rssMB‚ 2712.90625) disk./.usagePercent‚19.59861710259766 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0.078125 disk.sda1.out‚0' disk.nvme0c0n1.out‚7201.65771484375 disk.sda3.out‚ 64.4140625' proc.memory.availableMB‚ 146.8515625 proc.cpu.threads‚17 disk.dm-1.out‚ 64.4140625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚16.558265686035156 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚99.10368919372559 cpu‚25.21808456759967% disk.nvme0n1.out‚7201.65771484375 disk.nvme0c0n1.in‚0.078125ÝMJ™“¼‚(:ˆ ÄëåÔéÙéÇ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸“A½‚(:¶ ÄëåÔŒâÀÉ disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 7263.7421875 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 7263.7421875$ memory_percent‚99.10521507263184 proc.cpu.threads‚17* disk./.usagePercent‚19.600136251305845 disk.sda3.in‚0 disk.dm-1.out‚ 64.94140625 disk.sda1.out‚0$ disk./.usageGB‚292.0561943054199 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 12.2880859375' proc.memory.availableMB‚ 146.6015625* proc.memory.percent‚16.558265686035156 disk.rbd0.in‚0 network.sent‚ 3.719599e+06 disk.sda3.out‚ 64.94140625 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 64.94140625 network.recv‚664352 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 proc.memory.rssMB‚ 2712.90625 disk.drbd2080.out‚0 disk.drbd1716.out‚0 cpu‚25.216730538511584 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚ 7263.7421875 disk.drbd2080.in‚0 disk.sda.in‚0jõq¼2¾- _wandbruntime‚1775  _runtime‚1775ÅG0EÈ¿Â _runtime‚1775.880332898 _step‚159 ' lr-AdamW/pg1‚0.00015204529458897906 ' lr-AdamW/pg2‚0.00015204529458897906 trainer/global_step‚1999 $ _timestamp‚1.788442058833112e+09<¸ÍÀÇ trainer/global_step‚1999 $ _timestamp‚1.788442058833112e+09 _runtime‚1775.880332898 _step‚159 ' lr-AdamW/pg1‚0.00015204529458897906 ' lr-AdamW/pg2‚0.00015204529458897906Ÿ~@!z2Á- _wandbruntime‚1775  _runtime‚1775Cû/Õ ÂèO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43309124 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156267843572736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1880507786240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30932187 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873315627727 trainer/global_step‚1999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154290339457024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10412175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20519488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1977687778304 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140126516057600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34915112 ?9DeviceStatsMonitor.on_t…U§€rain_batch_start/num_alloc_retries‚0 _runtime‚1775.881286267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43286016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142006240719360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1880507786240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34915110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156268052007936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34915112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34915110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138221215093083 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140126516057600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34915112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1775.881286267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43286016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142006240719360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1880507786240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34915110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156268052007936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34915112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34915110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138221215093083 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1880507786240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30932187 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873315627727 trainer/global_step‚1999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154290339457024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 ’Å*“2Ä- _wandbruntime‚1776  _runtime‚17764–Ó°÷&ÅñM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884420596286223e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34932150 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34932152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156344715818496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1873458383375 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30947103 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142076600086528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚161 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10417117 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43307350 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154366046419968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142075799778304 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136415515494400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138289755078739 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78239500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1880652147200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20529477 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154366021647360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136415498455040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78239500 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1978483652096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78262636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210519040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1881435284992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140195147631104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30946594 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78262636 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140195164801536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 1776.67642946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142076600086528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1880652147200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10417623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43330484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34932152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚140195164801536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20529480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138288956838415 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1881435284992 trainer/global_step‚1999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140195147631104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78262636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210519040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1881435284992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140195147631104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30946594 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78262636 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140195164801536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 1776.67642946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142076600086528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1880652147200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10417623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43330484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34932152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚140195164801536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20529480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138288956838415 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1881435284992 trainer/global_step‚1999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140195147631104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚161 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10417117 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43307350 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154366046419968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142075799778304 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136415515494400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964¡æ²³Ù2Ç- _wandbruntime‚1776  _runtime‚1776†!5DȾ /loss/FRt2/assignment_loss‚4.224521462733929 /loss/FRt3/assignment_loss‚5.042997653143747 1loss/SRqqt2/assignment_loss‚2.388581206773718 .loss/FRt2/detection_loss‚2.356491062694635 0loss/SRqqt2/detection_loss‚2.443060094118118 _runtime‚1776.677575353 2loss/SRqqt1/assignment_loss‚2.8259841546867834 0loss/FBt3/assignment_loss‚0.6278915359742112 0loss/FBt4/assignment_loss‚0.7048402293949039 /loss/FRt1/detection_loss‚2.9570554196834564 /loss/FBt3/detection_loss‚3.4726591838730707 $ _timestamp‚1.788442059630431e+09 2loss/SRqqt4/assignment_loss‚2.2947467981266376 1loss/SRqqt4/detection_loss‚1.0864511818445028 /loss/FBt1/detection_loss‚3.2585372924804688 %loss/total_loss‚62.06032643814909 epoch‚1 0loss/FRt1/assignment_loss‚5.3652084412113314 1loss/FBt2/assignment_loss‚0.17686001757264797 /loss/SRqqt3/assignment_loss‚2.7562849566792 loss/FBt1/assignment_loss‚0 /loss/FRt3/detection_loss‚2.7582472937447684 0loss/SRqqt3/detection_loss‚2.628001848856608 /loss/FRt4/detection_loss‚1.9724069584661454 trainer/global_step‚1999 1loss/SRqqt1/detection_loss‚2.6094122149727563 0loss/FRt4/assignment_loss‚3.8396927296225702 /loss/FBt2/detection_loss‚1.0354570932528504 .loss/FBt4/detection_loss‚5.234937608242035 _step‚162÷eEIIÉà 0loss/FRt4/assignment_loss‚3.8396927296225702 /loss/FBt2/detection_loss‚1.0354570932528504 .loss/FBt4/detection_loss‚5.234937608242035 _step‚162 /loss/FRt2/assignment_loss‚4.224521462733929 /loss/FRt3/assignment_loss‚5.042997653143747 1loss/SRqqt2/assignment_loss‚2.388581206773718 .loss/FRt2/detection_loss‚2.356491062694635 0loss/SRqqt2/detection_loss‚2.443060094118118 _runtime‚1776.677575353 2loss/SRqqt1/assignment_loss‚2.8259841546867834 0loss/FBt3/assignment_loss‚0.6278915359742112 0loss/FBt4/assignment_loss‚0.7048402293949039 /loss/FRt1/detection_loss‚2.9570554196834564 /loss/FBt3/detection_loss‚3.4726591838730707 $ _timestamp‚1.788442059630431e+09 2loss/SRqqt4/assignment_loss‚2.2947467981266376 1loss/SRqqt4/detection_loss‚1.0864511818445028 /loss/FBt1/detection_loss‚3.2585372924804688 %loss/total_loss‚62.06032643814909 epoch‚1 0loss/FRt1/assignment_loss‚5.3652084412113314 1loss/FBt2/assignment_loss‚0.17686001757264797 /loss/SRqqt3/assignment_loss‚2.7562849566792 loss/FBt1/assignment_loss‚0 /loss/FRt3/detection_loss‚2.7582472937447684 0loss/SRqqt3/detection_loss‚2.628001848856608 /loss/FRt4/detection_loss‚1.9724069584661454 trainer/global_step‚1999 1loss/SRqqt1/detection_loss‚2.6094122149727563¢€{øΞÊ‚(:“ ÓëåÔ¾Ü©Ç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.481' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.696200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áB6Ë‚(:« ÓëåÔˆ±ÖÉ network.recv‚668579 disk.rbd1.out‚ 12.2880859375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.out‚ 7308.9375 disk.drbd2080.out‚0 network.sent‚ 3.72542e+06 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 7308.9375 disk.drbd2265.out‚0 proc.memory.rssMB‚ 2712.90625 cpu‚25.215344723812745 disk.nvme0c0n1.out‚ 7308.9375 disk.nvme0c0n1.in‚0.078125 disk.drbd2080.in‚0 disk.sda3.out‚ 65.52734375 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda.out‚ 65.52734375# disk./.usageGB‚292.070011138916 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 65.52734375 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚99.10082817077637' proc.memory.availableMB‚ 147.3203125* proc.memory.percent‚16.558265686035156 disk.sda.in‚0 disk.sda1.in‚0* disk./.usagePercent‚19.601063510594862 disk.rbd1.in‚0ÖéÃGžÌ‚(:“ âëåÔÔĵÇ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.229' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.245800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ß¡¦JÍ‚(:¿ âëåÔÁõ›É disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125! disk.dm-0.out‚7351.0087890625 disk.drbd2266.out‚0* proc.memory.percent‚16.558265686035156 proc.cpu.threads‚17 network.sent‚ 3.833984e+06$ disk.nvme0n1.out‚7351.0087890625$ memory_percent‚99.10082817077637 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚675525$ disk./.usageGB‚292.0794677734375 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0* disk./.usagePercent‚19.601698153272242 disk.rbd1.out‚ 12.4755859375 disk.sda3.out‚ 65.96484375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚7351.0087890625 disk.dm-1.out‚ 65.96484375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.memory.rssMB‚ 2712.90625 disk.sda.out‚ 65.96484375 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 cpu‚25.213999692916822 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0' proc.memory.availableMB‚ 147.3203125/3ÂW“΂(:ˆ ñëåÔ¨è®Ç gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒêèLÏ‚(:Á ñëåÔ­íÄÉ% disk./.usageGB‚292.08767318725586 disk.dm-1.out‚ 66.6328125 disk.rbd1.out‚ 12.4755859375 disk.sda1.out‚0 disk.sda.out‚ 66.6328125$ memory_percent‚99.10085201263428 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 66.6328125( proc.memory.availableMB‚ 147.31640625 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda.in‚0 network.recv‚679035 disk.drbd2080.out‚0 disk.rbd1.in‚0 cpu‚25.212654494768092 network.sent‚ 3.839209e+06* disk./.usagePercent‚19.602248825478394$ disk.nvme0n1.out‚7390.9423828125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda3.in‚0* proc.memory.percent‚16.558289527893066 proc.cpu.threads‚17 disk.dm-0.in‚0.078125! disk.dm-0.out‚7390.9423828125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚7390.9423828125# proc.memory.rssMB‚ 2712.91015625=4‡Ò2Ð- _wandbruntime‚1818  _runtime‚1818›ˆtóÆÑÀ & lr-AdamW/pg1‚0.0001517860709657223 & lr-AdamW/pg2‚0.0001517860709657223 trainer/global_step‚2049 $ _timestamp‚1.788442101943851e+09 _runtime‚1818.991079646 _step‚163vàTäËÒÅ & lr-AdamW/pg1‚0.0001517860709657223 & lr-AdamW/pg2‚0.0001517860709657223 trainer/global_step‚2049 $ _timestamp‚1.788442101943851e+09 _runtime‚1818.991079646 _step‚163£IC˜2Ó- _wandbruntime‚1818  _runtime‚1818-æJÃí'ÔçO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44379176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1926113347584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145485008375296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223115264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21019088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35767112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143558111903232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143558094732800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44379176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2026504677376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143558094732800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 $ _timestamp‚1.788442101944667e+09 _runtime‚1818.992151188 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1926896472064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21019085 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158074448973824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139689587151872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10662958 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1926896472064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35767110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2026703020032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145484208080384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35767112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1918745838379 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80123178 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31682043 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160101176766464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1919527039135 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1926113347584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10663500 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141609114191007 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139689570112512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35767110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264uÄÌò'ÕìO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1926896472064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21019085 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158074448973824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139689587151872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10662958 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1926896472064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35767110 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2026703020032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145484208080384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35767112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1918745838379 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80123178 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31682043 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160101176766464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1919527039135 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1926113347584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10663500 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141609114191007 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139689570112512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35767110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44379176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1926113347584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145485008375296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223115264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21019088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35767112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143558111903232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143558094732800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44379176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2026504677376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143558094732800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 $ _timestamp‚1.788442101944667e+09 _runtime‚1818.992151188 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¤ÑÕU2Ö- _wandbruntime‚1819  _runtime‚1819„Ö.Î×òM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚Ì=ŸO*&0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚165 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35784152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚145554567447552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1927040833024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21029077 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139756387901440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚145554567447552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31697067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2027666355712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44377402 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158150155936768 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143626726306304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80161552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1919669794783 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143626743476736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10667990 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884421027690835e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31697604 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80184688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚145553767139328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1920450995747 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚2049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139756387901440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚145554567447552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31697067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2027666355712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44377402 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158150155936768 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143626726306304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80161552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1919669794783 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143626743476736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10667990 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884421027690835e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31697604 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80184688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚145553767139328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1920450995747 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚2049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚1861.858032487 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1972502033408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36619112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45449228 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32431674 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45449228 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148962975736320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32432201 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144995416808699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10913513 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146989690578432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36619110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 _step‚168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163933954170368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1973285157888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163934162605568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146989673408000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21518688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148962175441408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143030476598272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45426120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2075395679744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10912989 trainer/global_step‚2099 % _timestamp‚1.7884421448105307e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36619112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146989690578432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146989673408000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148962175441408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161858558490624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144996215048815 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1965738450543 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36619110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1973285157888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143030459558912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82045230 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972502033408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161858583263232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82068340 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2075579342336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45426120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21518685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264G±4éó'çíO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45426120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2075395679744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10912989 trainer/global_step‚2099 % _timestamp‚1.7884421448105307e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36619112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146989690578432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146989673408000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148962175441408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161858558490624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144996215048815 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1965738450543 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36619110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1973285157888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143030459558912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82045230 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972502033408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161858583263232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82068340 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2075579342336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45426120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21518685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚1861.858032487 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1972502033408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36619112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45449228 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32431674 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45449228 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148962975736320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32432201 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144995416808699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10913513 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146989690578432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36619110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183662592 _step‚168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163933954170368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1973285157888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163934162605568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146989673408000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21518688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148962175441408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143030476598272¨Zb·2è- _wandbruntime‚1862  _runtime‚1862YÑü,÷&éñM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21528680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164010597749760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21528677 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147058322151936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161934265453568 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147058304981504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45470588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10918456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149031734500352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32446597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143097277347840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147058322151936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10917920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1862.661649133 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45470588 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198329344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2076555398144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36636152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1974212656640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164010820851712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143097294387200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚2099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1974212656640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82106740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82106740 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149032534808576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149032534808576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82083604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36636150 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240324 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147058304981504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45470588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10918456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149031734500352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32446597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143097277347840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147058322151936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10917920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636150 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1862.661649133 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45470588 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198329344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2076555398144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36636152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1974212656640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1640108208517 ¸_~1 12 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143097294387200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚2099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1974212656640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82106740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82106740 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149032534808576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149032534808576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82083604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36636150©..Ú2ë- _wandbruntime‚1862  _runtime‚1862¸y•[ìÕ /loss/FRt1/detection_loss‚2.6070880943383927 /loss/FRt2/detection_loss‚2.2545823380351067 0loss/FRt2/assignment_loss‚3.4313511461019517 epoch‚1 2loss/SRqqt1/assignment_loss‚2.4767702392169406 2loss/SRqqt3/assignment_loss‚2.9169299370712705 0loss/FBt4/assignment_loss‚0.5947550866338942 1loss/SRqqt4/detection_loss‚1.0140125362512966 /loss/FBt2/detection_loss‚0.9004224543326927 /loss/FBt3/detection_loss‚2.7426008019182415 /loss/FRt1/assignment_loss‚5.404096224369147 /loss/FRt3/assignment_loss‚5.742704416143483 0loss/SRqqt2/detection_loss‚2.436844680843682 0loss/SRqqt3/detection_loss‚2.465313793884383 /loss/FBt4/detection_loss‚3.9418267938825817 _step‚170 2loss/SRqqt2/assignment_loss‚1.7266805304920878 2loss/SRqqt4/assignment_loss‚2.3509875108798344 /loss/FBt1/assignment_loss‚0.838264000415802 .loss/FRt3/detection_loss‚3.026511724652915 &loss/total_loss‚59.987417842427305 % _timestamp‚1.7884421456154625e+09 0loss/FRt4/assignment_loss‚3.6928156789143882 0loss/FBt3/assignment_loss‚0.6723831885566243 /loss/FRt4/detection_loss‚1.8244273469845453 /loss/FBt1/detection_loss‚4.1346009254455565 _runtime‚1862.662655061 trainer/global_step‚2099 1loss/FBt2/assignment_loss‚0.27898856829605734 1loss/SRqqt1/detection_loss‚2.5124598247664314h|â`íÚ trainer/global_step‚2099 1loss/FBt2/assignment_loss‚0.27898856829605734 1loss/SRqqt1/detection_loss‚2.5124598247664314 /loss/FRt1/detection_loss‚2.6070880943383927 /loss/FRt2/detection_loss‚2.2545823380351067 0loss/FRt2/assignment_loss‚3.4313511461019517 epoch‚1 2loss/SRqqt1/assignment_loss‚2.4767702392169406 2loss/SRqqt3/assignment_loss‚2.9169299370712705 0loss/FBt4/assignment_loss‚0.5947550866338942 1loss/SRqqt4/detection_loss‚1.0140125362512966 /loss/FBt2/detection_loss‚0.9004224543326927 /loss/FBt3/detection_loss‚2.7426008019182415 /loss/FRt1/assignment_loss‚5.404096224369147 /loss/FRt3/assignment_loss‚5.742704416143483 0loss/SRqqt2/detection_loss‚2.436844680843682 0loss/SRqqt3/detection_loss‚2.465313793884383 /loss/FBt4/detection_loss‚3.9418267938825817 _step‚170 2loss/SRqqt2/assignment_loss‚1.7266805304920878 2loss/SRqqt4/assignment_loss‚2.3509875108798344 /loss/FBt1/assignment_loss‚0.838264000415802 .loss/FRt3/detection_loss‚3.026511724652915 &loss/total_loss‚59.987417842427305 % _timestamp‚1.7884421456154625e+09 0loss/FRt4/assignment_loss‚3.6928156789143882 0loss/FBt3/assignment_loss‚0.6723831885566243 /loss/FRt4/detection_loss‚1.8244273469845453 /loss/FBt1/detection_loss‚4.1346009254455565 _runtime‚1862.662655061ªð..“î‚(:ˆ ­ìåÔÆ—ºÈ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ ÿHï‚(:½ ­ìåÔÕ“ÀÊ% disk.nvme0n1.out‚7568.37255859375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2712.91015625 network.recv‚698376 disk.sda.out‚ 68.9140625 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.sda3.in‚0" disk.dm-0.out‚7568.37255859375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7568.37255859375# memory_percent‚99.1034746170044* proc.memory.percent‚16.558289527893066 network.sent‚ 3.964923e+06 cpu‚25.207231675872862 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 68.9140625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 68.9140625 disk.drbd2080.out‚0 disk.rbd1.out‚12.78125 disk.sda1.in‚0( proc.memory.availableMB‚ 146.88671875* disk./.usagePercent‚19.605945318265288$ disk./.usageGB‚292.1427536010742 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda2.out‚0>’Xåžð‚(:“ ¼ìåÔ‹éÃÇ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.367399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Й~Kñ‚(:À ¼ìåÔûÈïÉ disk.sda.out‚ 69.54296875% disk.nvme0n1.out‚7632.15087890625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17% disk./.usageGB‚292.16303634643555 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0# memory_percent‚99.1034746170044 disk.rbd0.out‚0 network.recv‚704929 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 69.54296875 disk.dm-1.out‚ 69.54296875* disk./.usagePercent‚19.607306510324847 disk.nvme0c0n1.in‚0.078125 disk.drbd2266.out‚0( proc.memory.availableMB‚ 146.88671875# proc.memory.rssMB‚ 2712.91015625 cpu‚25.20601775136546 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚7632.15087890625 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125* proc.memory.percent‚16.558289527893066' disk.nvme0c0n1.out‚7632.15087890625 network.sent‚ 4.073175e+06 disk.rbd1.out‚12.84375 disk.sda2.in‚0 ã·2ò- _wandbruntime‚1904  _runtime‚1904YDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150421269253632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150421269253632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150421252083200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167767023674368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165642668007424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146371366044672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152440142802432 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2011168661195 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2018890719232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37471112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37471112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚1904.949388439 _step‚172 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83967282 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22018285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152440142802432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33182643 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2011949861951 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2018890719232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37471110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152440943097344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146371349005312 % _timestamp‚1.7884421879017904e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46519280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11164871 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46519280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165642692780032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148383315906623 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148382517666507 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592ç;QÆó'÷íO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37471110 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152440142802432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33182643 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2011949861951 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2018890719232 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37471110 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152440943097344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146371349005312 % _timestamp‚1.7884421879017904e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46519280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11164871 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46519280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165642692780032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148383315906623 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148382517666507 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46496172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11164358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152440943097344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150421269253632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150421269253632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150421252083200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167767023674368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165642668007424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146371366044672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152440142802432 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2011168661195 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2018890719232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37471112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37471112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 _runtime‚1904.949388439 _step‚172 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83967282 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22018285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33197623 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84005656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37488150 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2012873818563 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150489883656704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148451057652163 $ _timestamp‚1.788442188713426e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46540640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187843584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_poo¸]Ð< l.freed‚ 2019818204672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150489883656704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33198130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150489900827136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167843901097472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212616192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165718350197760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11169850 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146438166794240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2012092617599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 trainer/global_step‚2149 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46517506 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167843688481280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37488150 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84028792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22028277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37488152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2019818204672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148450259411839 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84028792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33197623 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84005656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37488150 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2012873818563 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150489883656704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148451057652163 $ _timestamp‚1.788442188713426e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46540640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187843584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2019818204672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150489883656704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33198130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150489900827136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167843901097472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212616192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165718350197760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11169850 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146438166794240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2012092617599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 trainer/global_step‚2149 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46517506 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167843688481280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37488150 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84028792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22028277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37488152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2019818204672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148450259411839 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84028792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38323112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153852847928832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194148352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153852847928832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153852830758400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47589332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22517888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47566224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153852830758400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85889334 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33933975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149712238451712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2058161273359 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155918910458368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151769618524315 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38323110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2065279405056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38323112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33934499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169426777524224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155918110163456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171599955969536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.frÄ;Éeed‚155918110163456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2066062529536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155918110163456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2066062529536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38323112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153852847928832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194148352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153852847928832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153852830758400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47589332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22517888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47566224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153852830758400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85889334 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33933975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149712238451712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2058161273359 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155918910458368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151769618524315 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38323110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2065279405056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38323112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33934499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169426777524224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155918110163456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171599955969536°nüÛ,2Š- _wandbruntime‚1948  _runtime‚1948Èî”âø&‹òM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85927708 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11421652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2174349528064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22527877 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33949020 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85950844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153921462331904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151838158509971 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85927708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171676627690496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155988469530624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2174167975936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 trainer/global_step‚2199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153921479502336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11421143 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149779073280000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155987669222400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47610692 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚177 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2058304029007 % _timestamp‚1.7884422318605888e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153921479502336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169502484487168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2066990028288 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85950844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151837360269647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1948.908347208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38340150 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38340152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592ú~ý&Œ÷M _runtime‚1948.908347208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38340150 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38340152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85927708 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11421652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2174349528064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22527877 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33949020 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85950844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153921462331904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151838158509971 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85927708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171676627690496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155988469530624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2174167975936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 trainer/global_step‚2199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153921479502336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11421143 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149779073280000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155987669222400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47610692 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚177 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2058304029007 % _timestamp‚1.7884422318605888e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153921479502336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169502484487168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2066990028288 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85950844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151837360269647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0±•=±62- _wandbruntime‚1948  _runtime‚1948˜WóDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159396077524480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87834496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157284409433600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2222380306432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48636276 _runtime‚1991.324317759 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2112451215360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34685092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11667604 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155157517622239 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155156719382123 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87811386 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2222190352384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48636276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23017488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23017485 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153053144937472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159396077524480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2104372684767 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111668090880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157284426604032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39175112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚2249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48659384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159396877819392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159396877819392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11667083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175433077393408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 _step‚180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175433292120064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173210911813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39175110 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111668090880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39175110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48659384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34684568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39175112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157284426604032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189954048 % _timestamp‚1.7884422742769096e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0›„¯Ûó'›íO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159396077524480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87834496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157284409433600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2222380306432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48636276 _runtime‚1991.324317759 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2112451215360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34685092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11667604 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155157517622239 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155156719382123 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87811386 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2222190352384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48636276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23017488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23017485 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153053144937472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159396077524480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2104372684767 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111668090880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157284426604032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39175112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 trainer/global_step‚2249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48659384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159396877819392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159396877819392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11667083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175433077393408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 _step‚180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175433292120064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173210911813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39175110 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111668090880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39175110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48659384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34684568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39175112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157284426604032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189954048 % _timestamp‚1.7884422742769096e+09´Åwé‹2œ- _wandbruntime‚1992  _runtime‚1992‡2Ø—òM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175509952785920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11672034 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173286569231360 % _timestamp‚1.7884422750875726e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159465636583424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39192152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87849760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48680744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2105296641379 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48680744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157353041007104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112595576320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112595576320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48657610 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚Hßa23134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11672556 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2113378714112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34700036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87872896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39192152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87872896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2223358781952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87849760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39192150 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153119962726400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157353058177536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34699511 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2104515440415ü Ðºý&ž÷M \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2113378714112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48657610 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192037888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2223166744064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159466436891648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39192150 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23027480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112595576320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112595576320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48657610 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11672556 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2113378714112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34700036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87872896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39192152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87872896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2223358781952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87849760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39192150 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153119962726400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157353058177536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34699511 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2104515440415 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175509952785920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11672034 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173286569231360 % _timestamp‚1.7884422750875726e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159465636583424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39192152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87849760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48680744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2105296641379 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48680744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157353041007104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 B¡bž¦‚(:“ ÃíåÔâ©“È gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚139.027' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.805399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:(¨C§‚(:¸ ÃíåÔö›ÂÊ disk.nvme0n1.in‚0.078125 disk.nvme0c0n1.in‚0.078125& disk.nvme0c0n1.out‚8332.8486328125 disk.rbd1.out‚ 13.634765625 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚8332.8486328125$ memory_percent‚98.26362133026123) disk./.usagePercent‚19.61442787190481 disk.sda.in‚0 disk.sda.out‚74.21875 cpu‚25.195488603736116% disk./.usageGB‚292.26914978027344 disk.sda3.in‚0$ disk.nvme0n1.out‚8332.8486328125( proc.memory.availableMB‚ 284.48828125 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 network.recv‚749857 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2712.91015625 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚74.21875 disk.dm-1.out‚74.21875 disk.drbd2266.out‚0* proc.memory.percent‚16.558289527893066 network.sent‚ 4.51326e+06 disk.drbd2266.in‚0 disk.drbd2265.out‚0q”Ø:2¨- _wandbruntime‚2034  _runtime‚2034ÂÚôÔÉ©Ã _runtime‚2034.368269822 _step‚183 ' lr-AdamW/pg1‚0.00015025742982049614 ' lr-AdamW/pg2‚0.00015025742982049614 trainer/global_step‚2299 % _timestamp‚1.7884423173210578e+09‡î[ÎªÈ trainer/global_step‚2299 % _timestamp‚1.7884423173210578e+09 _runtime‚2034.368269822 _step‚183 ' lr-AdamW/pg1‚0.00015025742982049614 ' lr-AdamW/pg2‚0.00015025742982049614·ãAp2«- _wandbruntime‚2034  _runtime‚2034w\êq´%¬æO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40027112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11917558 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160715988108800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49706328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162874845180416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158544618480047 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162874044885504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162874044885504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49729436 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 # _timestamp‚1.78844231732194e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160716005279232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2158056776704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89756548 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _step‚184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2158839901184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49706328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35435176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160715988108800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11918088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2158839901184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2271353535488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _runtime‚2034.369611184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179266160139264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2150584096175 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40027112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11917558 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160715988108800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49706328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162874845180416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158544618480047 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162874044885504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162874044885504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49729436 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 # _timestamp‚1.78844231732194e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160716005279232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2158056776704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89756548 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _step‚184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2158839901184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49706328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35435176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160715988108800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11918088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2158839901184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2271353535488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 _runtime‚2034.369611184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179266160139264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2150584096175 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160784619682304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11922477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40044150 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179342801562112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158611561985263 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2158984262144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158612360225587 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 _runtime‚2035.192365699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23527077 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2272321143296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162944404252672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35450098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2158984262144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35449554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160784619682304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2159767399936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40044152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89771812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2150726851823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179343024664064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156460852172800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40044150 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11923018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788442318144548e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89771812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162944404252672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2159767399936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40044152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783137792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2272122813952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177070703520768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160784636852736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160784636852736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89794948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49727662 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177070678748160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89794948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49727662 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177070678748160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160784619682304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11922477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40044150 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179342801562112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158611561985263 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2158984262144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158612360225587 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 _runtime‚2035.192365699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23527077 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2272321143296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162944404252672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35450098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2158984262144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35449554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160784619682304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2159767399936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40044152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89771812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2150726851823¹Ñ¸ž2±- _wandbruntime‚2035  _runtime‚2035c­#§S²Í 1loss/FBt1/assignment_loss‚0.24698866624385118 /loss/FRt1/detection_loss‚2.5576541335024734 /loss/FRt2/detection_loss‚1.9534160188502736 /loss/FRt2/assignment_loss‚3.571735324131118 1loss/SRqqt1/assignment_loss‚2.179925718589833 .loss/FBt1/detection_loss‚2.795884758234024 /loss/FBt2/detection_loss‚0.9168849174246355 /loss/FRt4/assignment_loss‚3.540253851373317 -loss/FRt4/detection_loss‚2.11913549167625 1loss/SRqqt3/detection_loss‚2.4214198733529737 1loss/SRqqt4/detection_loss‚1.1314875200011973 trainer/global_step‚2299 0loss/FRt3/assignment_loss‚4.7438041642308235 0loss/FBt2/assignment_loss‚0.2899112599057242 0loss/FBt4/assignment_loss‚0.3654970611320702 /loss/FRt3/detection_loss‚2.5228183016180994 1loss/SRqqt1/detection_loss‚2.3475490855543235 _runtime‚2035.193485974 _step‚186 .loss/FBt3/detection_loss‚2.840919640660286 .loss/FBt4/detection_loss‚3.180514549215635 epoch‚1 0loss/SRqqt2/detection_loss‚2.154760954054919 % _timestamp‚1.7884423181463761e+09 1loss/SRqqt2/assignment_loss‚2.312409918981068 0loss/FBt3/assignment_loss‚0.6791627466678619 /loss/FRt1/assignment_loss‚5.015584296368538 1loss/SRqqt3/assignment_loss‚2.179351286542031 1loss/SRqqt4/assignment_loss‚2.219872491465787 %loss/total_loss‚54.28694202977711 3Õ%õ³Ò 1loss/SRqqt2/assignment_loss‚2.312409918981068 0loss/FBt3/assignment_loss‚0.6791627466678619 /loss/FRt1/assignment_loss‚5.015584296368538 1loss/SRqqt3/assignment_loss‚2.179351286542031 1loss/SRqqt4/assignment_loss‚2.219872491465787 %loss/total_loss‚54.28694202977711 1loss/FBt1/assignment_loss‚0.24698866624385118 /loss/FRt1/detection_loss‚2.5576541335024734 /loss/FRt2/detection_loss‚1.9534160188502736 /loss/FRt2/assignment_loss‚3.571735324131118 1loR4Pcss/SRqqt1/assignment_loss‚2.179925718589833 .loss/FBt1/detection_loss‚2.795884758234024 /loss/FBt2/detection_loss‚0.9168849174246355 /loss/FRt4/assignment_loss‚3.540253851373317 -loss/FRt4/detection_loss‚2.11913549167625 1loss/SRqqt3/detection_loss‚2.4214198733529737 1loss/SRqqt4/detection_loss‚1.1314875200011973 trainer/global_step‚2299 0loss/FRt3/assignment_loss‚4.7438041642308235 0loss/FBt2/assignment_loss‚0.2899112599057242 0loss/FBt4/assignment_loss‚0.3654970611320702 /loss/FRt3/detection_loss‚2.5228183016180994 1loss/SRqqt1/detection_loss‚2.3475490855543235 _runtime‚2035.193485974 _step‚186 .loss/FBt3/detection_loss‚2.840919640660286 .loss/FBt4/detection_loss‚3.180514549215635 epoch‚1 0loss/SRqqt2/detection_loss‚2.154760954054919 % _timestamp‚1.7884423181463761e+09ºò¶êž´‚(:“ ÒíåÔ°ÐÝÇ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.295' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.458999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãZ@µ‚(:µ ÒíåÔÄÄ÷É disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚98.26700687408447 network.sent‚ 4.519623e+06" disk.nvme0c0n1.out‚ 8386.921875 disk.sda.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 8386.921875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 cpu‚25.194446882208133 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda.out‚ 74.61328125 disk.sda2.out‚0 disk.rbd1.out‚ 13.759765625* proc.memory.percent‚16.558289527893066) disk./.usagePercent‚19.61495627145548$ disk./.usageGB‚292.2770233154297 disk.nvme0c0n1.in‚0.078125( proc.memory.availableMB‚ 283.93359375# proc.memory.rssMB‚ 2712.91015625 network.recv‚755026 disk.sda3.out‚ 74.61328125 disk.dm-1.in‚0 disk.dm-1.out‚ 74.61328125 disk.nvme0n1.out‚ 8386.921875 disk.drbd2080.in‚0 disk.rbd1.in‚0o­7ž¶‚(:“ áíåÔˆö˜Ç gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.123' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.624599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xó{…E·‚(:º áíåÔÄŽ°É disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.19329980086344 disk.dm-1.out‚ 75.21484375 disk.nvme0n1.in‚0.078125) disk./.usagePercent‚19.61598516573559% disk./.usageGB‚292.29235458374023 disk.rbd1.out‚ 13.908203125 disk.sda.out‚ 75.21484375$ memory_percent‚98.26548099517822 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2712.9140625 disk.nvme0c0n1.in‚0.078125" disk.nvme0n1.out‚ 8436.01953125 network.recv‚760179 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 8436.01953125 disk.sda.in‚0 disk.sda3.out‚ 75.21484375 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 8436.01953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 284.18359375 network.sent‚ 4.627118e+06 disk.drbd2265.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0* proc.memory.percent‚16.558313369750977 disk.sda1.in‚0 disk.sda1.out‚0Š®ÆMž¸‚(:“ ðíåÔñ—¯Ç gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.935' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.186999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³˜Ø|M¹‚(: ðíåÔ­·³Ê disk.rbd1.in‚0 disk.sda.out‚ 75.72265625 disk.dm-1.in‚0 disk.dm-1.out‚ 75.72265625 disk.drbd2266.out‚0 network.sent‚ 4.632244e+06 disk.sda1.in‚0# proc.memory.rssMB‚ 2712.91796875 cpu‚25.192248851485484% disk./.usageGB‚292.30765533447266 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚0.078125! disk.dm-0.out‚8519.1806640625& disk.nvme0c0n1.out‚8519.1806640625 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚763836* disk./.usagePercent‚19.617012011955428 disk.rbd1.out‚ 14.001953125 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 75.72265625$ disk.nvme0n1.out‚8519.1806640625 disk.drbd2265.out‚0' proc.memory.availableMB‚ 284.4140625 disk.drbd2080.out‚0 disk.sda.in‚0$ memory_percent‚98.26407432556152* proc.memory.percent‚16.558337211608887¤Í“'2º- _wandbruntime‚2077  _runtime‚2077`ÿ4êÇ»Á & lr-AdamW/pg2‚0.0001499056429973869 trainer/global_step‚2349 % _timestamp‚1.7884423605859456e+09 _runtime‚2077.633167276 _step‚187 & lr-AdamW/pg1‚0.0001499056429973869¥¯ÏÌ¼Æ _step‚187 & lr-AdamW/pg1‚0.0001499056429973869 & lr-AdamW/pg2‚0.0001499056429973869 trainer/global_step‚2349 % _timestamp‚1.7884423605859456e+09 _runtime‚2077.633167276»_ ù=2½- _wandbruntime‚2077  _runtime‚2077R«Êèî'¾èO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166352812541440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12168637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36185878 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12169190 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2320232064512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159734923830272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40879110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 trainer/global_step‚2349 _runtime‚2077.634203212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196795507583 % _timestamp‚1.7884423605867386e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40879112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166352012246528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2204445462528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40879112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2320046304768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164147583954432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2205228587008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2204445462528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2205228587008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164147566784000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36185322 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159734906790912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50799488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180779130847232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164147583954432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚188 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24016688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166352012246528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161930921097739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40879110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183099362911744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91678600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24016685 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2196014306827 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50799488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28215 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210532352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183099152379392 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23108 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36185878 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12169190 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2320232064512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159734923830272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40879110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 trainer/global_step‚2349 _runtime‚2077.634203212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196795507583 % _timestamp‚1.7884423605867386e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40879112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166352012246528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2204445462528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40879112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2320046304768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164147583954432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225929 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23110 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2205228587008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2204445462528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2205228587008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164147566784000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36185322 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159734906790912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28215 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50799488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180779130847232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164147583954432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚188 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24016688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166352012246528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161930921097739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40879110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183099362911744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91678600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24016685¼œŒHÒ2À- _wandbruntime‚2078  _runtime‚2078aã{~ÁñM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91717000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40896150 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183175796616192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 $ _timestamp‚1.788442361419561e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50797714 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180854788264960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2196938263231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40896152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24026680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12173695 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 299642368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40896152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91717000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2321204583424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2321008351232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚189 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183176017620992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40896150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50820848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36200935 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159801741619200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164216215527936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24026677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12174255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180854813037568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2205372947968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166421571305472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164216215527936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚2349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 QBDeviceSnp½ytatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36200372 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚2078.467400364 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2206156085760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159801724579840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164216198357504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164216198357504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166422371613696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166422371613696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91717000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2321204583424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2321008351232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚189 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183176017620992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40896150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50820848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36200935 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159801741619200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164216215527936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783137792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24026677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12174255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180854813037568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2205372947968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166421571305472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164216215527936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27966 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚2349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28215 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36200372 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚2078.467400364 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2206156085760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159801724579840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164216198357504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164216198357504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166422371613696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166422371613696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40896150 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183175796616192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196232192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 $ _timestamp‚1.788442361419561e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50797714 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180854788264960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2196938263231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40896152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24026680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12173695 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0½èûј2Ã- _wandbruntime‚2078  _runtime‚2078+u‰bXÄÒ 0loss/SRqqt1/detection_loss‚2.462398558070785 0loss/SRqqt2/detection_loss‚2.301889166945503 /loss/FRt1/detection_loss‚2.8004233100835014 .loss/FRt2/detection_loss‚2.263098270547661 0loss/FRt2/assignment_loss‚3.2909811721606688 .loss/FRt3/detection_loss‚2.921363023019606 /loss/FBt3/detection_loss‚2.4162222295999527 epoch‚1 trainer/global_step‚2349 2loss/SRqqt2/assignment_loss‚2.1174002279128348 /loss/FBt4/detection_loss‚2.8396804224361074 $ _timestamp‚1.788442361421399e+09 _step‚190 1loss/SRqqt3/detection_loss‚2.4973262089329795 0loss/FBt4/assignment_loss‚0.9082128703540465 /loss/FRt1/assignment_loss‚4.492101620225346 1loss/SRqqt3/assignment_loss‚2.577478141398043 1loss/FBt1/assignment_loss‚0.45173875201079583 .loss/FRt4/detection_loss‚2.123805372742936 2loss/SRqqt1/assignment_loss‚2.7839318106048987 /loss/FRt3/assignment_loss‚5.027750607459776 0loss/FRt4/assignment_loss‚3.6053767260164022 2loss/SRqqt4/assignment_loss‚2.2922102490015197 1loss/FBt2/assignment_loss‚0.28094244668376217 /loss/FBt1/detection_loss‚3.1680499580171375 /loss/FBt2/detection_loss‚0.9845745080825105 _runtime‚2078.468528083 0loss/FBt3/assignment_loss‚0.8315406561480183 1loss/SRqqt4/detection_loss‚1.2382226219041306 %loss/total_loss‚56.67671893035892Ô|]Å× 1loss/SRqqt3/detection_loss‚2.4973262089329795 0loss/FBt4/assignment_loss‚0.9082128703540465 /loss/FRt1/assignment_loss‚4.492101620225346 1loss/SRqqt3/assignment_loss‚2.577478141398043 1loss/FBt1/assignment_loss‚0.45173875201079583 .loss/FRt4/detection_loss‚2.123805372742936 2loss/SRqqt1/assignment_loss‚2.7839318106048987 /loss/FRt3/assignment_loss‚5.027750607459776 0loss/FRt4/assignment_loss‚3.6053767260164022 2loss/SRqqt4/assignment_loss‚2.2922102490015197 1loss/FBt2/assignment_loss‚0.28094244668376217 /loss/FBt1/detection_loss‚3.1680499580171375 /loss/FBt2/detection_loss‚0.9845745080825105 _runtime‚2078.468528083 0loss/FBt3/assignment_loss‚0.8315406561480183 1loss/SRqqt4/detection_loss‚1.2382226219041306 %loss/total_loss‚56.67671893035892 0loss/SRqqt1/detection_loss‚2.462398558070785 0loss/SRqqt2/detection_loss‚2.301889166945503 /loss/FRt1/detection_loss‚2.8004233100835014 .loss/FRt2/detection_loss‚2.263098270547661 0loss/FRt2/assignment_loss‚3.2909811721606688 .loss/FRt3/detection_loss‚2.921363023019606 /loss/FBt3/detection_loss‚2.4162222295999527 epoch‚1 trainer/global_step‚2349 2loss/SRqqt2/assignment_loss‚2.1174002279128348 /loss/FBt4/detection_loss‚2.8396804224361074 $ _timestamp‚1.788442361421399e+09 _step‚190¾Õ=´“Æ‚(:ˆ ÿíåÔǨÜÇ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C…£IÇ‚(:¾ ÿíåÔœæÉ$ memory_percent‚97.89955615997314( proc.memory.availableMB‚ 344.13671875* proc.memory.percent‚16.558337211608887 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 76.26171875 disk.nvme0n1.in‚0.078125 disk.rbd0.in‚0 disk.sda.out‚ 76.26171875 cpu‚25.191134429542053* disk./.usagePercent‚19.616358680728048$ disk./.usageGB‚292.2979202270508 disk.sda3.in‚0 disk.rbd1.out‚ 14.064453125 disk.sda1.out‚0% disk.nvme0c0n1.out‚8571.962890625 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.recv‚767764 disk.sda2.out‚0# disk.nvme0n1.out‚8571.962890625 disk.dm-0.out‚8571.962890625 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2712.91796875 network.sent‚ 4.63803e+06 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.dm-1.out‚ 76.26171875 disk.drbd2080.out‚0 ßBl‘È‚(:† ŽîåÔŒûûÇ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÝŸHÉ‚(:½ ŽîåÔŸœåÉ disk.drbd1716.out‚0" disk.dm-0.out‚8624.33837890625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 76.796875 disk.dm-1.in‚0$ disk./.usageGB‚292.3136787414551 disk.drbd2080.in‚0( proc.memory.availableMB‚ 343.98046875 disk.rbd1.out‚ 14.283203125 disk.nvme0n1.in‚0.078125! proc.memory.rssMB‚ 2712.921875 cpu‚25.190138881093425* disk./.usagePercent‚19.617416247851992 disk.sda1.in‚0' disk.nvme0c0n1.out‚8624.33837890625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚8624.33837890625 network.sent‚ 4.746666e+06 disk.rbd0.out‚0* proc.memory.percent‚16.558361053466797 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 76.796875 disk.drbd2266.out‚0$ memory_percent‚97.90050983428955 network.recv‚774981 disk.sda2.in‚0 disk.dm-1.out‚ 76.796875 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚0.078125{ù5žÊ‚(:“ îåÔÁ½˜Ç gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.895799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬[‘LË‚(:Á îåÔÚ›¥Ê disk.sda2.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚8673.07275390625 disk.drbd2265.in‚0 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2712.921875* proc.memory.percent‚16.558361053466797 disk.sda.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚19.61809108371225 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 14.345703125# memory_percent‚97.8992223739624 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 77.47265625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 network.sent‚ 4.752234e+06 disk.rbd0.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 344.19140625 proc.cpu.threads‚17 network.recv‚778611 disk.sda1.in‚0 disk.sda3.out‚ 77.47265625 disk.dm-1.out‚ 77.47265625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.18914497726874% disk./.usageGB‚292.32373428344727% disk.nvme0n1.out‚8673.07275390625' disk.nvme0c0n1.out‚8673.07275390625tâV2Ì- _wandbruntime‚2120  _runtime‚2120D½ ÈÍ _step‚191 ' lr-AdamW/pg1‚0.00014953866692879534 ' lr-AdamW/pg2‚0.00014953866692879534 trainer/global_step‚2399 % _timestamp‚1.7884424038636317e+09 _runtime‚ 2120.91086371Nª–âÍÎÇ _step‚191 ' lr-AdamW/pg1‚0.00014953866692879534 ' lr-AdamW/pg2‚0.00014953866692879534 trainer/global_step‚2399 % _timestamp‚1.7884424038636317e+09 _runtime‚ 2120.91086371¿n{2Ï- _wandbruntime‚2120  _runtime‚2120ÏÎÿ8í'ÐçO trainer/global_step‚2399 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124480 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2242225718235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167579162629632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169829979607552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51846432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167579162629632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2250834148352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24516285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 _step‚192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41731112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36936905 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165318820195663 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12421146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202143744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 177371136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51846432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51869540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41731110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51869540 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186932186759168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2368971167744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169830779902464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184563240364032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 $ _timestamp‚1.788442403864516e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36937434 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186932388902912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93577542 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2250834148352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169829979607552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163075796237312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12420620 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2242225718235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167579162629632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169829979607552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51846432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167579162629632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2250834148352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24516285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030857 _step‚192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800294912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27966 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41731112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36936905 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165318820195663 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12421146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202143744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 177371136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51846432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51869540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 299642368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41731110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51869540 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800294912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186932186759168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2368971167744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169830779902464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23110 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184563240364032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 $ _timestamp‚1.788442403864516e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36937434 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186932388902912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93577542 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2250834148352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169829979607552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163075796237312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12420620 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639052 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163142631065600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163142614026240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167647794203136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2243149674639 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51890900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36951905 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24526280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165385763700879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169900338974720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51867766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2251761633792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41748150 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2243930875603 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2252544771584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41748152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2369949619712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24526277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12426151 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167647777032704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51890900 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185746432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 2121.73730232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167647777032704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93615916 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚193 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167647794203136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187009057920512‹r¡ü&ÔöM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2251761633792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41748150 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2243930875603 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2252544771584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41748152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2369949619712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24526277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12426151 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167647777032704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51890900 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185746432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 2121.73730232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167647777032704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93615916 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚193 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167647794203136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187009057920512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41748150 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36952431 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639052 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163142631065600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163142614026240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167647794203136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225929 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030857 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2243149674639 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51890900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36951905 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24526280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165385763700879 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169900338974720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51867766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Áºá‚ò2Õ- _wandbruntime‚2121  _runtime‚2121cÀ-[ÖÕ 1loss/FBt4/assignment_loss‚0.42952581850840943 1loss/SRqqt4/detection_loss‚1.3571299822981824 epoch‚1 1loss/SRqqt1/assignment_loss‚2.576398561830106 2loss/SRqqt4/assignment_loss‚2.2585747330305503 /loss/FRt4/detection_loss‚2.2040786242485044 .loss/FBt1/detection_loss‚5.467466235160828 _runtime‚2121.738493428 _step‚194 1loss/SRqqt1/detection_loss‚2.3191151605999987 /loss/FBt2/detection_loss‚0.7101681758145874 /loss/FRt1/detection_loss‚2.7691676542162895 /loss/FRt2/detection_loss‚1.7691457855204742 0loss/FRt2/assignment_loss‚2.9428650786479316 2loss/SRqqt2/assignment_loss‚1.6127655753136978 1loss/FBt3/assignment_loss‚0.47327732957935287 1loss/SRqqt3/detection_loss‚2.4199442761284966 0loss/FRt1/assignment_loss‚5.7862242394023475 .loss/FRt3/detection_loss‚3.049749195575714 trainer/global_step‚2399 %loss/total_loss‚58.84904912310518 % _timestamp‚1.7884424046913588e+09 /loss/FRt3/assignment_loss‚4.780859533945719 1loss/SRqqt3/assignment_loss‚2.665585058076041 1loss/FBt2/assignment_loss‚0.19338082274443444 1loss/SRqqt2/detection_loss‚2.2812876623609792 /loss/FBt3/detection_loss‚2.9019404758106577 .loss/FBt4/detection_loss‚4.024572744965553 0loss/FRt4/assignment_loss‚3.1803620076179504 /loss/FBt1/assignment_loss‚0.675464391708374¼7&`×Ú 0loss/FRt2/assignment_loss‚2.9428650786479316 2loss/SRqqt2/assignment_loss‚1.6127655753136978 1loss/FBt3/assignment_loss‚0.47327732957935287 1loss/SRqqt3/detection_loss‚2.4199442761284966 0loss/FRt1/assignment_loss‚5.7862242394023475 .loss/FRt3/detection_loss‚3.049749195575714 trainer/global_step‚2399 %loss/total_loss‚58.84904912310518 % _timestamp‚1.7884424046913588e+09 /loss/FRt3/assignment_loss‚4.780859533945719 1loss/SRqqt3/assignment_loss‚2.665585058076041 1loss/FBt2/assignment_loss‚0.19338082274443444 1loss/SRqqt2/detection_loss‚2.2812876623609792 /loss/FBt3/detection_loss‚2.9019404758106577 .loss/FBt4/detection_loss‚4.024572744965553 0loss/FRt4/assignment_loss‚3.1803620076179504 /loss/FBt1/assignment_loss‚0.675464391708374 1loss/FBt4/assignment_loss‚0.42952581850840943 1loss/SRqqt4/detection_loss‚1.3571299822981824 epoch‚1 1loss/SRqqt1/assignment_loss‚2.576398561830106 2loss/SRqqt4/assignment_loss‚2.2585747330305503 /loss/FRt4/detection_loss‚2.2040786242485044 .loss/FBt1/detection_loss‚5.467466235160828 _runtime‚2121.738493428 _step‚194 1loss/SRqqt1/detection_loss‚2.3191151605999987 /loss/FBt2/detection_loss‚0.7101681758145874 /loss/FRt1/detection_loss‚2.7691676542162895 /loss/FRt2/detection_loss‚1.7691457855204742Â/8ÿžØ‚(:“ ¬îåÔ˜Ù©Ç gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.191' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.038199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cš?ËÇÙ‚(:Á ¬îåÔ£ä¯É network.sent‚ 4.758184e+06 disk.sda3.out‚ 77.79296875$ memory_percent‚97.90287017822266 network.recv‚783097 disk.sda3.in‚0 disk.drbd1716.in‚0% pØTp…roc.memory.availableMB‚ 343.59375* proc.memory.percent‚16.558361053466797 cpu‚25.188083213844166 proc.cpu.threads‚17* disk./.usagePercent‚19.618431573732792% disk.nvme0n1.out‚8725.01220703125 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0% disk./.usageGB‚292.32880783081055 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 14.345703125 disk.drbd1716.out‚0" disk.dm-0.out‚8725.01220703125 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.dm-1.out‚ 77.79296875 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚8725.01220703125 disk.sda.out‚ 77.79296875! proc.memory.rssMB‚ 2712.921875Uܳ’Ú‚(:‡ »îåÔ› ëÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚192.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|eºùKÛ‚(:À »îåÔ¾€óÊ disk.drbd2265.in‚0 disk.sda.in‚0 network.sent‚ 4.866612e+06 network.recv‚789450% disk./.usageGB‚292.34102630615234 disk.drbd1716.out‚0% disk.nvme0n1.out‚8773.87158203125 disk.sda1.in‚0( proc.memory.availableMB‚ 336.55078125 disk.sda3.in‚0 disk.dm-1.out‚ 78.3515625 disk.dm-0.in‚0.078125 disk.sda3.out‚ 78.3515625 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚8773.87158203125 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 14.3779296875 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0) proc.memory.percent‚16.60141944885254 cpu‚25.20263731583754 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2080.out‚0$ memory_percent‚97.94585704803467) disk./.usagePercent‚19.61925156586496 disk.drbd2080.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚8773.87158203125 disk.sda.out‚ 78.3515625" proc.memory.rssMB‚ 2719.9765625åuÓf“Ü‚(:ˆ ÊîåÔúûÀÇ gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü@€ŸCÝ‚(:¸ ÊîåÔ“ßÊ disk.rbd1.out‚ 14.5029296875 disk.sda2.in‚0# proc.memory.availableMB‚336.375" proc.memory.rssMB‚ 2719.9765625 cpu‚25.233386101199134 disk.dm-0.out‚8834.130859375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚ 78.83984375 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0$ disk./.usageGB‚292.3473815917969 disk.sda3.in‚0 disk.dm-1.out‚ 78.83984375 disk.sda.out‚ 78.83984375$ memory_percent‚97.94692993164062 network.recv‚794350 disk.nvme0c0n1.in‚0.078125) proc.memory.percent‚16.60141944885254 proc.cpu.threads‚17 network.sent‚ 4.87292e+06 disk.drbd1716.in‚0# disk.nvme0n1.out‚8834.130859375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚8834.130859375* disk./.usagePercent‚19.619678074417006 disk.drbd2266.in‚0ã"Þ‚(:’ ÙîåÔ®Æ¥É gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.409' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.681800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÁ¦9ß‚(:® ÙîåÔિË disk.dm-1.out‚79.5625 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚8870.650390625# proc.memory.availableMB‚336.125 network.recv‚797790* disk./.usagePercent‚19.620183177282062 disk.nvme0n1.in‚0.078125 disk.sda.out‚79.5625 cpu‚25.263012190777324% disk./.usageGB‚292.35490798950195 disk.rbd0.in‚0" proc.memory.rssMB‚ 2719.9765625 disk.dm-0.out‚8870.650390625 network.sent‚ 4.878152e+06 disk.drbd2265.in‚0 disk.sda3.out‚79.5625$ memory_percent‚97.94845581054688 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.5654296875 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚8870.650390625 disk.drbd2265.out‚0 disk.drbd2080.out‚0) proc.memory.percent‚16.60141944885254 proc.cpu.threads‚17 disk.drbd2266.out‚01¯!å2à- _wandbruntime‚2188  _runtime‚2188n©Ã!U*áÏT SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 605071872 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 12013120000 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 11048832512 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚173381642868224 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚17170432 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚177 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 585986640 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12259950592 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚95335055 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2406825910272 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1010 trainer/global_step‚2417 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚171093933142528 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚173382247940096 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚180 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2288314797568 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚188260132639744 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚42697262 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚171093915972096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18279 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚52637795 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚37857136 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚188260157412352 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10672704512 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚521 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2287726896128 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚171093915972096 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚173382247940096 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166680064 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚173381642868224 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚190666958550016 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1021187072 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95353336 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚25115332 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18279 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚52637795 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 587901440 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27966 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1051 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1010‘ƒDoZ*âÔT UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚42697262 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚171093915972096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18279 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚52637795 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚37857136 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚188260157412352 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10672704512 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚521 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2287726896128 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚171093915972096 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚173382247940096 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166680064 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚173381642868224 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚190666958550016 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1021187072 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95353336 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚25115332 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18279 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚52637795 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 587901440 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27966 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1051 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1010 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 605071872 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 12013120000 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 11048832512 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚173381642868224 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚17170432 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚177 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 585986640 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12259950592 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚95335055 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2406825910272 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1010 trainer/global_step‚2417 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚171093933142528 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚173382247940096 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚180 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2288314797568 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚188260132639744 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2ÕÓؽ2ã- _wandbruntime‚2189  _runtime‚2189ÆÔ8‹äÙR H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚95355947 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚173427754648576 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚171139050910208 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 12013120000 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18279 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚171139033739776 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚521 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12259950592 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2407356758528 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚521 % _timestamp‚1.7884424726304882e+09 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2288115836928 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28215 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚173427149576704 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚17170432 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1021187072 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚25123686 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚166483751727104 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2279949305847 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚42710372 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚173427754648576 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 605071872 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18281 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 trainer/global_step‚2417 _step‚196 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚25123683 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 605071872 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2288115836928 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚521 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1021187072 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚12746542 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚42710372 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚180 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚95374228 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚52663856 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587901440 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚37870225 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚190713448622080 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚24772608 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2288703738368 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚52663856 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚166483734687744 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28215 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2407214851072 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27966 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚168763098006951 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚2189.678082294 ]ODeviceStatsMonitor.on_validation_batch_end/iÛtþÔnactive_split_bytes.large_pool.peak‚ 172325376 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019225929 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27966 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚168763701032951 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1010 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18281 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18279 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚42710370 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166680064 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11635030857 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2288703738368 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚42710370 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371895808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28215 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2407214851072 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27966 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚168763098006951 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚2189.678082294 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019225929 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27966 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚168763701032951 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1010 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18281 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18279 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚42710370 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166680064 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11635030857 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2288703738368 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚42710370 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371895808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3488464653491974 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.29004162549972534 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666865348816 particle/accuracy_5_of_6‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 _step‚197 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48361581563949585 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09118176996707916 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05363984778523445 +jet/accuracy_2_of_2‚0.07610692828893661 particle/accuracy_3_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.25925925374031067 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2483401596546173 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0052910055965185165 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.27932098507881165 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13264724612236023 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2697056233882904 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2558102011680603 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9909257292747498 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3508620858192444 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2478979229927063 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0769713744521141 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.29747214913368225 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 particle/accuracy_5_of_5‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30596569180488586 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0517241396009922 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9935441017150879 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24203097820281982 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44861114025115967 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.06439393758773804 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.011010978370904922 particle/accuracy_6_of_7‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013492063619196415 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0182027667760849 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.979957103729248 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2589229643344879 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38843217492103577 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2934512495994568 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714291036129 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1053343266248703 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21551908552646637 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_3‚0.728874146938324 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.01515151560306549 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2778193950653076 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6171622276306152 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *particle/specificity‚0.946190357208252 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07220854610204697 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16157318651676178 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.43375787138938904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.09459452331066132 jet/accuracy_4_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31465744972229004 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4W@¥WSRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.303916871547699 particle/accuracy_4_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3928973078727722 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.788442486529723e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.33015963435173035 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.3822955787181854 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6574504971504211 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9470142126083374 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0067007276229560375 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.017258932814002037 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.02380952425301075 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4690476953983307 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0067793638445436954 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.25381943583488464 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9862217307090759 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 *jet/accuracy_2_of_6‚0.3589743375778198 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2433723360300064 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3295087516307831 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.05592992156744003 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4078467786312103 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27197444438934326 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05585446581244469 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9879561066627502 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41878265142440796 ,jet/accuracy_4_of_5‚0.008064515888690948 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14327752590179443 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2476053535938263 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9919412732124329 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 +jet/accuracy_2_of_5‚0.37876981496810913 &particle/f_score‚0.337822824716568 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.21574077010154724 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9768891334533691 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9979325532913208 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3663279116153717 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 6 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.229166641831398 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2549024522304535 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.287001371383667 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05585446581244469 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0541829876601696 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08004534989595413 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2825861871242523 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27403274178504944 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38656938076019287 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9694601893424988 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.28568118810653687 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11149424314498901 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_3‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08666667342185974 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9889456033706665 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12996304035186768 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09166565537452698 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9731969237327576 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11186905950307846 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006778098177164793 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18953976035118103 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11522546410560608 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.613636314868927 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.397916704416275 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.01315789483487606 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9831575155258179 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13237661123275757 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8401370048522949 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9798044562339783 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35488665103912354 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02005331590771675 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.46057769656181335 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.320175439119339 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3295087516307831 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3413786292076111 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.003921568859368563 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *loss/val_total_loss‚56.479418835185584 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.377424955368042 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.002788800746202469 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08652273565530777 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10360508412122726 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9911462068557739 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.32902300357818604 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3573912978172302 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.27030977606773376 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 epoch‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2614136040210724 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_4_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9820159077644348 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19195400178432465 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 particle/accuracy_2_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07750879228115082 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.41342732310295105 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11052896827459335 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1805555522441864 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2889294922351837 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9873563051223755 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953007102012634 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9833052158355713 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31488099694252014 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09567491710186005 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_1_of_2‚0.6243652701377869 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9206350445747375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9621508717536926 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.05365600064396858 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14846733212471008 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12852001190185547 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1041509136557579 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 jet/accuracy_4_of_6‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 particle/accuracy_3_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3245586156845093 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.26245924830436707 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4093755781650543 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044294122606515884 *jet/accuracy_2_of_3‚0.1745130866765976 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2522927522659302 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10935672372579575 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 +jet/accuracy_3_of_5‚0.08597030490636826 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3215872645378113 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09231363981962204 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10321363806724548 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.29758787155151367 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2831307053565979 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.054088838398456573 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.25196778774261475 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2611394226551056 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9298029541969299 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9962584972381592 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.26893940567970276 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8394097685813904 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007833721116185188 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09186232835054398 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3251916766166687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4864055812358856 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32471683621406555 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1862623542547226 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36103686690330505 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003421229310333729 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt3SRqqt0FBt/event_purity‚0wµš›ä(ê§ :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38656938076019287 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9694601893424988 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.28568118810653687 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11149424314498901 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_3‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08666667342185974 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9889456033706665 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12996304035186768 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09166565537452698 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9731969237327576 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11186905950307846 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006778098177164793 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18953976035118103 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11522546410560608 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.613636314868927 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.397916704416275 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.01315789483487606 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9831575155258179 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13237661123275757 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8401370048522949 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9798044562339783 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35488665103912354 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02005331590771675 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.46057769656181335 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.320175439119339 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3295087516307831 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3413786292076111 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.003921568859368563 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *loss/val_total_loss‚56.479418835185584 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.377424955368042 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.002788800746202469 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08652273565530777 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10360508412122726 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9911462068557739 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.32902300357818604 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3573912978172302 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.27030977606773376 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 epoch‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2614136040210724 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_4_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9820159077644348 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19195400178432465 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 particle/accuracy_2_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07750879228115082 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.41342732310295105 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11052896827459335 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1805555522441864 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2889294922351837 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9873563051223755 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953007102012634 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9833052158355713 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31488099694252014 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09567491710186005 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_1_of_2‚0.6243652701377869 &ÛN0M³j Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9206350445747375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9621508717536926 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.05365600064396858 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14846733212471008 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12852001190185547 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1041509136557579 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 jet/accuracy_4_of_6‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 particle/accuracy_3_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3245586156845093 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.26245924830436707 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4093755781650543 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044294122606515884 *jet/accuracy_2_of_3‚0.1745130866765976 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2522927522659302 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10935672372579575 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 +jet/accuracy_3_of_5‚0.08597030490636826 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3215872645378113 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09231363981962204 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10321363806724548 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.29758787155151367 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2831307053565979 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.054088838398456573 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.25196778774261475 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2611394226551056 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9298029541969299 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9962584972381592 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.26893940567970276 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8394097685813904 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007833721116185188 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09186232835054398 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3251916766166687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4864055812358856 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32471683621406555 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1862623542547226 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36103686690330505 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003421229310333729 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.49343323707580566 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.985041081905365 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37552595138549805 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02500215917825699 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.046107079833745956 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0014662756584584713 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1259259283542633 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8050567507743835 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.03641752526164055 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.07692307978868484 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1962369978427887 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 trainer/global_step‚2417 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1273168921470642 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2530864179134369 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3125000298023224 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.993013322353363 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13226088881492615 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05349794402718544 particle/accuracy_1_of_6‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39474236965179443 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0747743621468544 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2633698582649231 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.010638297535479069 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1365809589624405 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1412731260061264 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9776270985603333 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 jet/accuracy_5_of_7‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3993839919567108 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9776313900947571 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3061102032661438 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3733333647251129 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.29594284296035767 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.06418918818235397 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666269302368 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9683951139450073 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24836598336696625 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953007102012634 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48901888728141785 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.32602113485336304 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2644781172275543 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06531144678592682 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0158565454185009 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_2_of_5‚1 *jet/accuracy_1_of_6‚0.8461538553237915 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2760147452354431 _runtime‚2203.579385707 -jet/accuracy_4_of_4‚0.0007215007208287716 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17060548067092896 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9113476276397705 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.046934451907873154 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25602275133132935 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4607827961444855 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2677820920944214 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3062015473842621 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9774520993232727 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.786424458026886 particle/accuracy_1_of_4‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9894018173217773 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9868161082267761 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3488464653491974 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.29004162549972534 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2291666865348816 particle/accuracy_5_of_6‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 _step‚197 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48361581563949585 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09118176996707916 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05363984778523445 +jet/accuracy_2_of_2‚0.07610692828893661 particle/accuracy_3_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.25925925374031067 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2483401596546173 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0052910055965185165 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.27932098507881165 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13264724612236023 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2697056233882904 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2558102011680603 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9909257292747498 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3508620858192444 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2478979229927063 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0769713744521141 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.29747214913368225 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 particle/accuracy_5_of_5‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30596569180488586 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0517241396009922 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9935441017150879 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24203097820281982 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44861114025115967 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.06439393758773804 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.011010978370904922 particle/accuracy_6_of_7‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013492063619196415 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0182027667760849 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.979957103729248 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2589229643344879 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38843217492103577 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2934512495994568 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714291036129 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1053343266248703 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21551908552646637 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_3‚0.728874146938324 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.01515151560306549 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2778193950653076 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6171622276306152 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *particle/specificity‚0.946190357208252 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07220854610204697 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16157318651676178 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.43375787138938904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.09459452331066132 jet/accuracy_4_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31465744972229004 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.303916871547699 particle/accuracy_4_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3928973078727722 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 $ _timestamp‚1.788442486529723e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.33015963435173035 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.3822955787181854 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6574504971504211 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9470142126083374 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0067007276229560375 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.017258932814002037 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.02380952425301075 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4690476953983307 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0067793638445436954 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.25381943583488464 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9862217307090759 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 *jet/accuracy_2_of_6‚0.3589743375778198 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2433723360300064 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3295087516307831 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.05592992156744003 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4078467786312103 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27197444438934326 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05585446581244469 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9879561066627502 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41878265142440796 ,jet/accuracy_4_of_5‚0.008064515888690948 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14327752590179443 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2476053535938263 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9919412732124329 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 +jet/accuracy_2_of_5‚0.37876981496810913 &particle/f_score‚0.337822824716568 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.21574077010154724 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9768891334533691 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9979325532913208 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3663279116153717 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 6 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.229166641831398 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2549024522304535 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.287001371383667 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05585446581244469 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0541829876601696 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08004534989595413 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2825861871242523 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27403274178504944 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0Å 9Û¹’ë‚(:‡ ÷îåÔç¾×Ç gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚34.0 gpu.0.powerWatts‚102.144' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4288 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö¤ãLì‚(:Á ÷îåÔõðÉ disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚ 80.71484375" disk.dm-0.out‚9002.46435546875 network.recv‚809362 disk.sda1.out‚0 disk.sda.out‚ 80.71484375 disk.drbd1716.out‚0 proc.memory.rssMB‚3208.875 network.sent‚ 4.992772e+06* disk./.usagePercent‚19.621231784142314 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0$ memory_percent‚99.73399639129639% disk.nvme0n1.out‚9002.46435546875 disk.drbd2266.in‚0' proc.memory.availableMB‚ 43.58203125 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚9002.46435546875 disk.dm-1.out‚ 80.71484375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.8076171875 cpu‚25.317669274659096 disk.drbd2265.in‚0% disk./.usageGB‚292.37053298950195 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚19.585418701171875Z.÷ƒ“í‚(:ˆ †ïåÔ™…ÃÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Äÿi@î‚(:µ †ïåÔ€ÀçÉ network.sent‚ 5.118427e+06 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚9625.220703125* proc.memory.percent‚16.649413108825684$ disk./.usageGB‚292.9309272766113 disk.sda3.in‚0 disk.drbd1716.in‚0# disk.nvme0n1.out‚9625.220703125 disk.drbd2265.out‚0$ memory_percent‚93.06724071502686 cpu‚25.312321789002425 proc.cpu.threads‚17* disk./.usagePercent‚19.658840314952347 disk.sda.out‚81.3125 network.recv‚816430 disk.sda3.out‚81.3125 disk.dm-1.out‚81.3125 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2727.83984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 15.2841796875 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1135.86328125 disk.dm-0.out‚9625.220703125 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚0.078125s‡È$“ï‚(:ˆ •ïåÔÛâ±Ç gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚190.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B\~CPð‚(:Å •ïåÔ¼ËÂÉ disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚10109.970703125! disk.dm-0.out‚10108.720703125 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 82.14453125 disk.drbd1716.out‚0 network.sent‚ 5.123661e+06$ memory_percent‚93.06576251983643* proc.memory.percent‚16.649413108825684 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚10109.970703125) proc.memory.availableMB‚ 1136.10546875 disk.rbd1.out‚ 15.3466796875 disk.sda1.out‚0 disk.dm-0.in‚0.078125# proc.memory.rssMB‚ 2727.83984375 cpu‚25.310698726822537 disk.rbd0.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚0.078125 network.recv‚819753 disk.sda3.out‚ 82.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 82.14453125* disk./.usagePercent‚19.659450124898907 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0% disk./.usageGB‚292.94001388549805 disk.sda2.in‚0 disk.sda2.out‚0x{ Š2ñ- _wandbruntime‚2236  _runtime‚2236÷pÚ³ÇòÁ _runtime‚2236.405101815 _step‚198 & lr-AdamW/pg1‚0.0001491565780919841 & lr-AdamW/pg2‚0.0001491565780919841 trainer/global_step‚2449 % _timestamp‚1.7884425193578243e+09Ñÿ*$ÌóÆ trainer/global_step‚2449 % _timestamp‚1.7884425193578243e+09 _runtime‚2236.405101815 _step‚198 & lr-AdamW/pg1‚0.0001491565780919841 & lr-AdamW/pg2‚0.0001491565780919841Æäâ U2ô- _wandbruntime‚2236  _runtime‚2236W± nõèO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2321743092736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2442527335936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38502027 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMoni}J ¯€&tor.on_train_batch_start/active_bytes.all.allocated‚176175686886400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173853143498240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2322526217728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚2449 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43409040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38502564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193694223713792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96817722 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2322526217728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12960029 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223114752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191251696377856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169111087303680 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176174886590976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171424727377037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542001 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53408682 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193694446828544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313640073357 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53431791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53431791 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176175686886400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43409040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176174886590976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2312858872597 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96840833 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 % _timestamp‚1.7884425193585608e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25541998 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2321743092736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173853160668672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12960563 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198342144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171423929136917 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173853160668672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173853143498240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169111070264320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191251721150464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176174886590976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171424727377037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542001 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53408682 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193694446828544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313640073357 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53431791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53431791 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176175686886400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43409040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176174886590976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2312858872597 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96840833 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 % _timestamp‚1.7884425193585608e+09Ç@žò 2÷- _wandbruntime‚2237  _runtime‚2237Ò cø&øòM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12965601 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53453151 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38517044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2314564029969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚2449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173921775071744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322670578176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12965054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173921792242176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173921775071744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323453716480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171491670882249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176245245958656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53430016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204620288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229392896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43426082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191327403340800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193770864393216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323453716480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96856096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25551990 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173921792242176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2443690445312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25551993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169177888053248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2313782829001 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53453151 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193771093786112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2443485825024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96879233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191327378568192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171492469122577 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169177905092608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96879233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43426080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322670578176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3è@ ù÷M D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96879233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43426080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322670578176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12965601 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53453151 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38517044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2314564029969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚2449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173921775071744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322670578176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12965054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/ac¤^ǽtive_bytes.large_pool.allocated‚173921792242176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173921775071744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323453716480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171491670882249 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176245245958656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53430016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204620288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229392896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43426082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191327403340800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193770864393216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323453716480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96856096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25551990 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173921792242176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2443690445312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25551993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169177888053248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2313782829001 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53453151 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193771093786112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2443485825024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96879233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191327378568192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171492469122577 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169177905092608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÈÂj¾s2ú- _wandbruntime‚2237  _runtime‚2237V´øOûÉ /loss/FRt3/assignment_loss‚4.378279447555542 0loss/FBt1/assignment_loss‚0.9374808056600159 0loss/FBt3/assignment_loss‚0.7267204821109772 0loss/SRqqt1/detection_loss‚2.155831587314606 %loss/total_loss‚56.29086729111441 /loss/FRt1/assignment_loss‚4.899545693397522 2loss/SRqqt3/assignment_loss‚2.2352340232048715 1loss/SRqqt4/assignment_loss‚2.054247448932041 .loss/FRt3/detection_loss‚2.931748390197754 /loss/FBt2/detection_loss‚0.8823332521639177 0loss/FBt2/assignment_loss‚0.3210334656703603 1loss/FBt4/assignment_loss‚0.17065600026398897 -loss/SRqqt2/detection_loss‚2.381229745845 .loss/FBt3/detection_loss‚2.429390008632953 epoch‚2 trainer/global_step‚2449 _runtime‚2237.252508442 /loss/FRt4/assignment_loss‚3.612311265047859 .loss/FRt4/detection_loss‚2.350953650825164 % _timestamp‚1.7884425202053216e+09 .loss/FRt2/detection_loss‚1.767063208998969 /loss/FRt2/assignment_loss‚3.362184312481147 2loss/SRqqt1/assignment_loss‚2.4022865017255146 /loss/FRt1/detection_loss‚2.9165315781320844 1loss/SRqqt3/detection_loss‚2.1120378105413344 .loss/FBt4/detection_loss‚3.330086998641491 .loss/FBt1/detection_loss‚5.081957459449768 _step‚201 2loss/SRqqt2/assignment_loss‚1.5802739982803662 0loss/SRqqt4/detection_loss‚1.271450156041167C‡FêTüÎ 0loss/SRqqt4/detection_loss‚1.271450156041167 /loss/FRt3/assignment_loss‚4.378279447555542 0loss/FBt1/assignment_loss‚0.9374808056600159 0loss/FBt3/assignment_loss‚0.7267204821109772 0loss/SRqqt1/detection_loss‚2.155831587314606 %loss/total_loss‚56.29086729111441 /loss/FRt1/assignment_loss‚4.899545693397522 2loss/SRqqt3/assignment_loss‚2.2352340232048715 1loss/SRqqt4/assignment_loss‚2.054247448932041 .loss/FRt3/detection_loss‚2.931748390197754 /loss/FBt2/detection_loss‚0.8823332521639177 0loss/FBt2/assignment_loss‚0.3210334656703603 1loss/FBt4/assignment_loss‚0.17065600026398897 -loss/SRqqt2/detection_loss‚2.381229745845 .loss/FBt3/detection_loss‚2.429390008632953 epoch‚2 trainer/global_step‚2449 _runtime‚2237.252508442 /loss/FRt4/assignment_loss‚3.612311265047859 .loss/FRt4/detection_loss‚2.350953650825164 % _timestamp‚1.7884425202053216e+09 .loss/FRt2/detection_loss‚1.767063208998969 /loss/FRt2/assignment_loss‚3.362184312481147 2loss/SRqqt1/assignment_loss‚2.4022865017255146 /loss/FRt1/detection_loss‚2.9165315781320844 1loss/SRqqt3/detection_loss‚2.1120378105413344 .loss/FBt4/detection_loss‚3.330086998641491 .loss/FBt1/detection_loss‚5.081957459449768 _step‚201 2loss/SRqqt2/assignment_loss‚1.5802739982803662É WÓý‚(:ˆ ¤ïåÔý­§Ç gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.307' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05äö2Fþ‚(:» ¤ïåÔîËÉ disk.dm-1.out‚82.625 disk.nvme0c0n1.in‚0.078125( disk.nvme0c0n1.out‚10290.41943359375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda3.out‚82.625& disk.nvme0n1.out‚10290.41943359375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.recv‚824151* disk./.usagePercent‚19.659584528854385 disk.sda.out‚82.625 disk.drbd1716.out‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1136.05078125# proc.memory.rssMB‚ 2727.83984375 disk.rbd1.in‚0 disk.sda3.in‚0 cpu‚25.308979068873462 network.sent‚ 5.129605e+06 disk.drbd2080.out‚0 disk.rbd1.out‚ 15.3466796875 disk.rbd0.in‚0$ disk./.usageGB‚292.9420166015625 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚10290.41943359375* proc.memory.percent‚16.649413108825684 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 proc.cpu.threads‚17 disk.sda.in‚0$ memory_percent‚93.06609630584717Í`½“ÿ‚(:ˆ ³ïåÔãÃÀÇ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚187.833' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ùíE€‚(:º ³ïåÔ†´É disk.sda3.out‚ 83.234375 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚10355.263671875 disk.sda2.out‚0 cpu‚25.30717862922222 network.sent‚ 5.238792e+06 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda.out‚ 83.234375 proc.cpu.threads‚17% disk./.usageGB‚292.96829986572266 disk.drbd1716.out‚0! disk.dm-0.out‚10355.263671875 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚10355.263671875 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0' proc.memory.availableMB‚ 1136.015625# proc.memory.rssMB‚ 2727.83984375 disk.dm-1.in‚0* proc.memory.percent‚16.649413108825684 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0* disk./.usagePercent‚19.661348420765297 disk.drbd2266.in‚0 disk.rbd1.out‚ 15.37890625 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 network.recv‚830416 disk.sda3.in‚0 disk.dm-1.out‚ 83.234375 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚93.06631088256836N³=ž‚(:“ ÂïåÔÚŠ·Ç gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.335' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.866999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xa C‚‚(:¸ ÂïåÔ÷¯ÃÉ disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚10419.1181640625 disk.dm-0.in‚0.078125 disk.rbd0.in‚0 disk.sda.out‚83.78125 proc.cpu.threads‚17 network.sent‚ 5.244977e+06 network.recv‚835064 disk.rbd1.out‚ 15.58984375 cpu‚25.30550303725516$ disk./.usageGB‚292.9807319641113 disk.sda3.out‚83.78125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚10419.1181640625 disk.sda.in‚0* proc.memory.percent‚16.649413108825684) disk./.usagePercent‚19.66218274931939 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1136.015625 disk.sda2.in‚0 disk.dm-1.out‚83.78125" disk.dm-0.out‚10419.1181640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚93.06631088256836# proc.memory.rssMB‚ 2727.83984375 disk.nvme0c0n1.in‚0.078125Œ¼Uß2ƒ- _wandbruntime‚2280  _runtime‚2280‚}÷Ç„Á trainer/global_step‚2499 % _timestamp‚1.7884425630445416e+09 _runtime‚2280.091778835 _step‚202 & lr-AdamW/pg1‚0.0001487594561136941 & lr-AdamW/pg2‚0.0001487594561136941Ñ.lÌ…Æ & lr-AdamW/pg1‚0.0001487594561136941 & lr-AdamW/pg2‚0.0001487594561136941 trainer/global_step‚2499 % _timestamp‚1.7884425630445416e+09 _runtime‚2280.091778835 _step‚202Ê%þ2†- _wandbruntime‚2280  _runtime‚2280Î í'‡çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26041598 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2359851484765 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195035830667264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172451959710720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚2499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197527299405312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2368914903552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177284739343872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44261040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39250290 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54501843 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44261042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177284722173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13208692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179652853952000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172451976750080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54478734 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2491276687360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174811029994725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2359070284005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788442563045365e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179653654247424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179652853952000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2368914903552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2368131778560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44261042 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195035805894656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54478734 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2491468738048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2359851484765 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 eQDeviceStatüElj sMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195035830667264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172451959710720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚2499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197527299405312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2368914903552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177284739343872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44261040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39250290 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54501843 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44261042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177284722173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13208692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179652853952000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172451976750080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54478734 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2491276687360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174811029994725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489Ëõ‰s»2‰- _wandbruntime‚2280  _runtime‚2280úø˜Tø&ŠòM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54500068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚204 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98801285 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39265748 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26051590 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179723213319680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13213619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54523203 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39265209 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2492439917568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197603733868032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13214155 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194134528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2360775441377 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884425638606477e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172518777499648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44278082 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177353370917376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98778148 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2359994240409 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98801285 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177353353746944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54500068 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174878771740057 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54523203 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2492245783040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174879569980385 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98778148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44278082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2280.908508285Ãzmý&‹÷M ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172518794539008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195111488084992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2369842402304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 trainer/global_step‚2499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98801285 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177353353746944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54500068 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174878771740057 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54523203 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2492245783040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174879569980385 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98778148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44278082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2280.908508285 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54500068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚204 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98801285 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39265748 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26051590 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179723213319680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13213619 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54523203 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39265209 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2492439917568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197603733868032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13214155 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194134528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2360775441377 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884425638606477e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172518777499648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44278082 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177353370917376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98778148 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2359994240409 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183130821313024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26541201 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55548786 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45113040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26541198 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45113042 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2415303589376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198819940184064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2540231219712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45113042 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55571895 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180716300848640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13459612 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180716318019072 % _timestamp‚1.7884426064019463e+09˜x•÷'˜ñO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183131621608448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2414520464384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180716318019072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178198130852533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183130821313024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2415303589376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13459086 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201360171403776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201359965066240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175792849157120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100661826 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183131621608448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2414520464384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100684937 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183130821313024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26541201 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55548786 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45113040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26541198 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45113042 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2415303589376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198819940184064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2540231219712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45113042 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55571895 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180716300848640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13459612 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180716318019072 % _timestamp‚1.7884426064019463e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180716300848640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178198929092653 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55548786 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚2323.450765235 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45113040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 _step‚207 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40000813 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198819915411456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100723337 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198895597601792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45130082 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2416231088128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40015750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100700200 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13464557 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178266670838193 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201436612084224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198895622374400 % _timestamp‚1.7884426072678301e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2406205651817 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180784932422144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100700200 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2416231088128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55570120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180784949592576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180784949592576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26551190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2541014482432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40015224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40015224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100723337 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198895597601792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45130082 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2416231088128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40015750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100700200 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13464557 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178266670838193 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201436612084224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198895622374400 % _timestamp‚1.7884426072678301e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2406205651817 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180784932422144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100700200 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2416231088128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55570120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180784949592576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180784949592576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26551190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2541014482432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010Ð" Ô˜2œ- _wandbruntime‚2324  _runtime‚2324Hž’PTÎ /loss/FBt4/assignment_loss‚0.494503853597962 0loss/FRt4/assignment_loss‚3.9694714790484946 /loss/FBt3/assignment_loss‚0.591106563720435 /loss/FRt4/detection_loss‚2.1826696249305226 .loss/FBt4/detection_loss‚2.997564627064599 _runtime‚2324.316683745 2loss/SRqqt3/assignment_loss‚2.4074956424332954 2loss/SRqqt4/assignment_loss‚2.0917583939073414 1loss/SRqqt4/detection_loss‚1.1277654230080802 0loss/FRt2/assignment_loss‚3.6687632468011646 0loss/FBt2/assignment_loss‚0.2577084770745712 0loss/SRqqt2/detection_loss‚2.149815993507703 % _timestamp‚1.7884426072695045e+09 /loss/FRt3/detection_loss‚2.9920247235605792 1loss/SRqqt1/detection_loss‚2.4745096242741536 0loss/SRqqt3/detection_loss‚2.486705221034385 _step‚209 /loss/FRt1/assignment_loss‚5.121315360069275 2loss/SRqqt1/assignment_loss‚2.2782730773875586 2loss/SRqqt2/assignment_loss‚2.0778997752815487 1loss/FBt1/assignment_loss‚0.11675805598497391 .loss/FRt1/detection_loss‚2.616068758898311 .loss/FRt2/detection_loss‚2.195972512827979 ,loss/FBt1/detection_loss‚8.5701003074646 %loss/total_loss‚62.03551313271751 /loss/FBt2/detection_loss‚0.9836678628284582 epoch‚2 /loss/FRt3/assignment_loss‚5.125411187448809 /loss/FBt3/detection_loss‚3.0581833405627146 trainer/global_step‚2549`s5YžÓ /loss/FBt4/assignment_loss‚0.494503853597962 0loss/FRt4/assignment_loss‚3.9694714790484946 /loss/FBt3/assignment_loss‚0.591106563720435 /loss/FRt4/detection_loss‚2.1826696249305226 .loss/FBt4/detection_loss‚2.997564627064599 _runtime‚2324.316683745 2loss/SRqqt3/assignment_loss‚2.4074956424332954 2loss/SRqqt4/assignment_loss‚2.0917583939073414 1loss/SRqqt4/detection_loss‚1.1277654230080802 0loss/FRt2/assignment_loss‚3.6687632468011646 0loss/FBt2/assignment_loss‚0.2577084770745712 0loss/SRqqt2/detection_loss‚2.149815993507703 % _timestamp‚1.7884426072695045e+09 /loss/FRt3/detection_loss‚2.9920247235605792 1loss/SRqqt1/detection_loss‚2.4745096242741536 0loss/SRqqt3/detection_loss‚2.486705221034385 _step‚209 /loss/FRt1/assignment_loss‚5.121315360069275 2loss/SRqqt1/assignment_loss‚2.2782730773875586 2loss/SRqqt2/assignment_loss‚2.0778997752815487 1loss/FBt1/assignment_loss‚0.11675805598497391 .loss/FRt1/detection_loss‚2.616068758898311 .loss/FRt2/detection_loss‚2.195972512827979 ,loss/FBt1/detection_loss‚8.5701003074646 %loss/total_loss‚62.03551313271751 /loss/FBt2/detection_loss‚0.9836678628284582 epoch‚2 /loss/FRt3/assignment_loss‚5.125411187448809 /loss/FBt3/detection_loss‚3.0581833405627146 trainer/global_step‚2549Ñ ·xM“Ÿ‚(:ˆ ïïåÔϯÆÇ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÐ@¬P ‚(:Å ïïåÔŠÑÜÉ disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚93.06693077087402 network.recv‚848438 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125# disk.dm-0.out‚10567.87939453125 disk.nvme0c0n1.in‚0.078125 cpu‚25.300608248049414 proc.cpu.threads‚17$ disk./.usageGB‚293.0165672302246 disk.sda3.in‚0 disk.rbd1.out‚ 15.6845703125* proc.memory.percent‚16.649413108825684& disk.nvme0n1.out‚10567.87939453125 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚10567.87939453125( proc.memory.availableMB‚ 1135.9140625 disk.dm-1.in‚0 disk.dm-1.out‚ 85.3515625 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 85.3515625 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 network.sent‚ 5.365231e+06 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 85.3515625 disk.sda2.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2727.83984375) disk./.usagePercent‚19.66458768409604 disk.drbd2265.in‚0XJh$“¡‚(:ˆ þïåÔ“«ÁÇ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.121' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07‘{K¢‚(:À þïåÔ¢•¢Ê disk.nvme0c0n1.in‚0.078125$ memory_percent‚93.06700229644775 proc.cpu.threads‚17 disk.dm-0.in‚0.078125 disk.rbd0.in‚0* proc.memory.percent‚16.649413108825684 disk.sda2.out‚0 disk.sda3.out‚ 85.8203125 disk.dm-1.in‚0 disk.dm-1.out‚ 85.8203125$ disk.nvme0n1.out‚10618.212890625 disk.drbd2266.in‚0 disk.rbd1.out‚ 15.8095703125 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚855579 network.sent‚ 5.474852e+06 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚10618.212890625# proc.memory.rssMB‚ 2727.83984375 disk.drbd1716.in‚0! disk.dm-0.out‚10618.212890625 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 85.8203125 cpu‚25.29900778071449 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1135.90234375* disk./.usagePercent‚19.665911755063135$ disk./.usageGB‚293.0362968444824 disk.drbd1716.out‚0Pò“£‚(:ˆ ðåÔ÷ÿÁÇ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Ò|éK¤‚(:À ðåÔ©›êÉ# proc.memory.rssMB‚ 2727.83984375 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.sda3.in‚0) disk./.usagePercent‚19.66678832486037 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 86.3828125 disk.nvme0n1.in‚0.078125 disk.sda.in‚0 disk.sda3.out‚ 86.3828125) proc.memory.availableMB‚ 1135.91015625 network.recv‚859125 disk.drbd1716.in‚0! disk.dm-0.out‚10659.794921875 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 86.3828125$ disk./.usageGB‚293.0493583679199 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 15.8720703125 disk.sda1.in‚0$ memory_percent‚93.06695461273193 cpu‚25.297448535435766$ disk.nvme0n1.out‚10659.794921875 disk.drbd2266.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚10659.794921875* proc.memory.percent‚16.649413108825684 network.sent‚ 5.480189e+06 disk.drbd2265.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125–Ïh2¥- _wandbruntime‚2368  _runtime‚2368*:ÅȦ trainer/global_step‚2599 $ _timestamp‚1.788442651411431e+09 _runtime‚2368.458705087 _step‚210 ' lr-AdamW/pg1‚0.00014792044688681524 ' lr-AdamW/pg2‚0.00014792044688681524×uá꤂ ' lr-AdamW/pg2‚0.00014792044688681524 trainer/global_step‚2599 $ _timestamp‚1.788442651411431e+09 _runtime‚2368.458705087 _step‚210 ' lr-AdamW/pg1‚0.00014792044688681524Òiƒ2¨- _wandbruntime‚2368  _runtime‚2368©?Jò'©ìO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2588996790784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45965042 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2461692275200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚2599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184147896694272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40750125 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2461692275200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202604049700864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 _step‚211 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200439296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40750675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186609588969472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2452274307581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184147879523840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179133738603520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2368.459622619 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184147896694272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45965040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184147879523840 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186608788674048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56641947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13709327 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179133755642880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56618838 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205193046491648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205192821279744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27040801 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181586029950461 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56641947 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2460909150208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181585231710341 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2451493106821 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27040798 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225211904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186609588969472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13709874 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102606989 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 % _timestamp‚1.7884426514121475e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2460909150208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45965040!:š—÷'ªñO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27040801 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181586029950461 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56641947 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2460909150208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181585231710341 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2451493106821 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27040798 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225211904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186609588969472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13709874 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102606989 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 % _timestamp‚1.7884426514121475e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2460909150208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45965040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2588996790784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45965042 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2461692275200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚2599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184147896694272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40750125 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2461692275200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202604049700864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 _step‚211 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200439296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40750675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186609588969472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2452274307581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184147879523840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179133738603520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2368.459622619 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184147896694272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45965040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184147879523840 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186608788674048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56641947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13709327 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179133755642880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56618838 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205193046491648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205192821279744Ó°zN2«- _wandbruntime‚2369  _runtime‚2369wë¬öM SEDeviceSh ©ë&tatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179200573431808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2589767867904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45982080 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2589964099584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184216528267776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13714284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56663307 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102645389 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181652973455673 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2452417063225 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2462619773952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102645389 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 % _timestamp‚1.7884426522733848e+09 _runtime‚2369.321413382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56640172 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184216511097344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40765624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27050790 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184216511097344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27050793 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186678347732992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102622252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚186679148041728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45982082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205269474986496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13714831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205269695990784 _step‚212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40765074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2599 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102622252 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45982082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56663307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2453198264193 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56640172 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202679731891200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0X¹[;'­ûM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184216511097344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27050793 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186678347732992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102622252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚186679148041728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45982082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205269474986496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13714831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205269695990784 _step‚212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40765074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2599 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102622252 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45982082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56663307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2453198264193 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56640172 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202679731891200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179200573431808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2589767867904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45982080 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2589964099584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184216528267776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13714284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56663307 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102645389 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181652973455673 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2452417063225 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2462619773952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102645389 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 % _timestamp‚1.7884426522733848e+09 _runtime‚2369.321413382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56640172 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184216511097344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40765624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27050790 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ô,ŒÑ‘2®- _wandbruntime‚2369  _runtime‚2369OW!BW¯Ñ /loss/FBt3/assignment_loss‚0.543532971658351 /loss/FRt1/detection_loss‚2.7069741800809517 /loss/FRt2/detection_loss‚2.2732751440060768 1loss/SRqqt2/detection_loss‚1.7884912166525335 %loss/total_loss‚53.53989705722874 /loss/FRt3/assignment_loss‚4.381985629477152 0loss/FRt4/assignment_loss‚2.9651927067683292 2loss/SRqqt1/assignment_loss‚2.3474556338928996 0loss/FBt2/assignment_loss‚0.3025300529861955 /loss/FRt3/detection_loss‚2.6789326856775983 .loss/FRt4/detection_loss‚1.820703231371366 1loss/SRqqt4/detection_loss‚1.2879251709038562 % _timestamp‚1.7884426522752864e+09 _runtime‚2369.322474665 0loss/SRqqt3/detection_loss‚2.474035827552571 trainer/global_step‚2599 0loss/FBt1/assignment_loss‚0.7664891630411148 0loss/FBt4/assignment_loss‚0.6499918525410399 /loss/FBt4/detection_loss‚2.8394777476787567 /loss/FRt1/assignment_loss‚4.505530320681059 1loss/SRqqt2/assignment_loss‚1.720687210012008 2loss/SRqqt4/assignment_loss‚2.0500901600327155 /loss/FBt1/detection_loss‚3.3728779355684915 _step‚213 1loss/SRqqt1/detection_loss‚2.2482919976824807 .loss/FBt3/detection_loss‚2.669114526361227 /loss/FRt2/assignment_loss‚3.983863956049869 2loss/SRqqt3/assignment_loss‚2.4172381618443657 /loss/FBt2/detection_loss‚0.7452095747077333 epoch‚2Y/Y\°Ö _runtime‚2369.322474665 0loss/SRqqt3/detection_loss‚2.474035827552571 trainer/global_step‚2599 0loss/FBt1/assignment_loss‚0.7664891630411148 0loss/FBt4/assignment_loss‚0.6499918525410399 /loss/FBt4/detection_loss‚2.8394777476787567 /loss/FRt1/assignment_loss‚4.505530320681059 1loss/SRqqt2/assignment_loss‚1.720687210012008 2loss/SRqqt4/assignment_loss‚2.0500901600327155 /loss/FBt1/detection_loss‚3.3728779355684915 _step‚213 1loss/SRqqt1/detection_loss‚2.2482919976824807 .loss/FBt3/detection_loss‚2.669114526361227 /loss/FRt2/assignment_loss‚3.983863956049869 2loss/SRqqt3/assignment_loss‚2.4172381618443657 /loss/FBt2/detection_loss‚0.7452095747077333 epoch‚2 /loss/FBt3/assignment_loss‚0.543532971658351 /loss/FRt1/detection_loss‚2.7069741800809517 /loss/FRt2/detection_loss‚2.2732751440060768 1loss/SRqqt2/detection_loss‚1.7884912166525335 %loss/total_loss‚53.53989705722874 /loss/FRt3/assignment_loss‚4.381985629477152 0loss/FRt4/assignment_loss‚2.9651927067683292 2loss/SRqqt1/assignment_loss‚2.3474556338928996 0loss/FBt2/assignment_loss‚0.3025300529861955 /loss/FRt3/detection_loss‚2.6789326856775983 .loss/FRt4/detection_loss‚1.820703231371366 1loss/SRqqt4/detection_loss‚1.2879251709038562 % _timestamp‚1.7884426522752864e+09Õä¾ùO“±‚(:ˆ œðåÔËàÖÇ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D¥›fS²‚(:È œðåÔ‘ùüÉ) disk./.usagePercent‚19.66807117861445 disk.drbd2266.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0# disk.dm-0.out‚10723.32568359375( proc.memory.availableMB‚ 1135.6640625 network.recv‚863801 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda.out‚ 86.87890625$ memory_percent‚93.06845664978027 cpu‚25.29600038391029 disk.drbd2265.in‚0 disk.rbd1.out‚ 16.2314453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125# proc.memory.rssMB‚ 2727.83984375% disk./.usageGB‚293.06847381591797 disk.rbd1.in‚0( disk.nvme0c0n1.out‚10723.32568359375 network.sent‚ 5.486234e+06 disk.sda.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚10723.32568359375* proc.memory.percent‚16.649413108825684 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 86.87890625 disk.dm-1.out‚ 86.87890625 proc.cpu.threads‚17hf„!“³‚(:ˆ «ðåÔ€ÃÐÇ gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O"BIS´‚(:È «ðåÔÏ°ÀÉ disk.sda.out‚ 87.40234375 disk.drbd1716.out‚0( proc.memory.availableMB‚ 1135.8359375 network.sent‚ 5.595187e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚10759.38525390625 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2727.83984375 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 87.40234375 disk.drbd2080.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚293.06222915649414 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚10759.38525390625 disk.drbd2266.out‚0 cpu‚25.294480383810747 proc.cpu.threads‚17 disk.dm-1.out‚ 87.40234375 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0$ memory_percent‚93.06740760803223* proc.memory.percent‚16.649413108825684 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.nvme0n1.in‚0.078125 network.recv‚870346( disk./.usagePercent‚19.6676520942809 disk.rbd1.in‚0 disk.rbd1.out‚ 16.2939453125( disk.nvme0c0n1.out‚10759.38525390625–~­ë“µ‚(:ˆ ºðåÔ‡ó¥Ç gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚190.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊZuS¶‚(:È ºðåÔÐÁÅÉ( disk.nvme0c0n1.out‚10820.91650390625$ memory_percent‚93.06895732879639 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2727.83984375 cpu‚25.292967887986126# disk.dm-0.out‚10820.91650390625 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 16.4189453125 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 network.sent‚ 5.60145e+06 network.recv‚875132$ disk./.usageGB‚293.0789375305176& disk.nvme0n1.out‚10820.91650390625) proc.memory.availableMB‚ 1135.58203125 proc.cpu.threads‚17 disk.dm-1.out‚ 87.83984375 disk.nvme0n1.in‚0.078125 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda.in‚0) disk./.usagePercent‚19.66877340728087 disk.sda3.out‚ 87.83984375 disk.dm-1.in‚0 disk.sda.out‚ 87.83984375 disk.sda2.in‚0* proc.memory.percent‚16.649413108825684&c!Ö2·- _wandbruntime‚2413  _runtime‚2413ŠÚÄ"ɸà trainer/global_step‚2649 % _timestamp‚1.7884426962151918e+09 _runtime‚2413.262397228 _step‚214 ' lr-AdamW/pg1‚0.00014747873448649171 ' lr-AdamW/pg2‚0.00014747873448649171‰ç êÎ¹È trainer/global_step‚2649 % _timestamp‚1.7884426962151918e+09 _runtime‚2413.262397228 _step‚214 ' lr-AdamW/pg1‚0.00014747873448649171 ' lr-AdamW/pg2‚0.00014747873448649171Ö¤—m¨2º- _wandbruntime‚2413  _runtime‚2413ù;ÎÔ»ìO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2498485718989 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187579458199040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2507297836032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2508080961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2508080961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41501003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚215 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190086756035072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209025868731904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884426962159166e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184973130808269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2637926337536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13960060 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184972332568149 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187579475369472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46817042 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209026085555200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57688890 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46817040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OFDeviceStatsMonitor.on_traiJ}‰/àn_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27540401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104505930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57711999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46817040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206388159217664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2413.263400692 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46817042 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41500458 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190086756035072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13960602 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 trainer/global_step‚2649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2637734286848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2497704518229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507297836032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190087556330496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206388134445056 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507297836032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190087556330496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206388134445056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2508080961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2508080961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41501003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚215 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190086756035072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209025868731904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884426962159166e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184973130808269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2637926337536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13960060 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184972332568149 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187579475369472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46817042 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209026085555200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57688890 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46817040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27540401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104505930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57711999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46817040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206388159217664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2413.263400692×oe'Í2½- _wandbruntime‚2414  _runtime‚2414æ祵ü&¾öM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187648106942976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185040872553809 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46834080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508225321472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2509008459776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104544304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57733359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13965513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41515369 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2638901527040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185040074313481 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 % _timestamp‚1.7884426970604877e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104567441 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190157115402752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190157115402752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187648089772544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41515906 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2498628474633 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46834082 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104567441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13964979 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46834080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27550393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2414.108359416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508225321472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209102742935040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57710224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27550390 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190156315094016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2499409675601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2509008459776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104544304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57733359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13965513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41515369 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2638901527040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185040074313481 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 % _timestamp‚1.7884426970604877e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104567441 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190157115402752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190157115402752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187648089772544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41515906 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2498628474633 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46834082 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104567441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13964979 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46834080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27550393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2414.108359416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2508225321472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209102742935040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57710224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27550390 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190156315094016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/acti{~) ve_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2499409675601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46834080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2508225321472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193565523691520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28040001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58782051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191011036874240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106451093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 trainer/global_step‚2699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212859244478976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191011054044672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2686975744512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2686789985280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _step‚219 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2544697130397 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106427982 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185815517496320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47669042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2553686521856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193564723396096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193565523691520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188359433425957 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185815534535680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543915929637 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¤·’÷'ÎñO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193564723396096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193565523691520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188359433425957 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185815534535680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543915929637 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 % _timestamp‚1.7884427412095025e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193564723396096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42250902 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191011054044672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47669042 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28039998 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193565523691520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28040001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58782051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191011036874240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106451093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 trainer/global_step‚2699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212859244478976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191011054044672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2686975744512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2686789985280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _step‚219 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2544697130397 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106427982 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185815517496320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47669042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2553686521856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47686082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚2699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210247950924800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106466356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47686080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58803411 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193635082763776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106489493 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚220 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42265827 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193635082763776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2687751226880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47686082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191079668447744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191079668447744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185882352324608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106489493 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212935900480512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28049990 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193634282455040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42266373 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2554614007296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14215837 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592s•:'ÑûM _step‚220 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42265827 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193635082763776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2687751226880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47686082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191079668447744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191079668447744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185882352324608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106489493 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212935900480512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28049990 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193634282455040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42266373 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2554614007296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14215837 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2555397145600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58780276 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188427175171289 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884427420755565e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188427973411617 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47686082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚2699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210247950924800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106466356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47686080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58803411 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193635082763776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106489493 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489Ü©·½-2Ò- _wandbruntime‚2459  _runtime‚2459.cÕTÓÎ epoch‚2 /loss/FRt3/assignment_loss‚4.260573524981737 0loss/FRt4/assignment_loss‚3.6379979807755043 /loss/FRt1/detection_loss‚2.6298443506925535 /loss/FRt3/detection_loss‚2.9700342528522015 1loss/SRqqt1/detection_loss‚2.6853062438964845 1loss/SRqqt4/detection_loss‚1.2488408813577727 1loss/SRqqt3/assignment_loss‚2.108382190125329 %loss/total_loss‚56.21347235760439 _step‚221 .loss/FBt1/detection_loss‚4.609753052393596 /loss/FBt2/detection_loss‚0.7347681431230553 1loss/SRqqt2/assignment_loss‚2.315272838498155 /loss/FBt3/assignment_loss‚0.607723370282786 -loss/FRt4/detection_loss‚2.32286078354408 /loss/FBt3/detection_loss‚2.4469729087182452 trainer/global_step‚2699 /loss/FRt1/assignment_loss‚4.167041106101794 1loss/SRqqt1/assignment_loss‚1.793498603105545 .loss/FRt2/detection_loss‚2.501527408533024 .loss/FBt4/detection_loss‚3.543661469221115 0loss/FRt2/assignment_loss‚3.5410891337828203 2loss/SRqqt4/assignment_loss‚2.2056028046602503 1loss/FBt4/assignment_loss‚0.40525355184799994 0loss/SRqqt2/detection_loss‚2.396424170051302 1loss/SRqqt3/detection_loss‚2.4446883056844984 _runtime‚2459.124859048 /loss/FBt1/assignment_loss‚0.287917032110272 1loss/FBt2/assignment_loss‚0.34843825126426964 % _timestamp‚1.7884427420776625e+09ø ÞYÔÓ epoch‚2 /loss/FRt3/assignment_loss‚4.260573524981737 0loss/FRt4/assignment_loss‚3.6379979807755043 /loss/FRt1/detection_loss‚2.6298443506925535 /loss/FRt3/detection_loss‚2.9700342528522015 1loss/SRqqt1/detection_loss‚2.6853062438964845 1loss/SRqqt4/detection_loss‚1.2488408813577727 1loss/SRqqt3/assignment_loss‚2.108382190125329 %loss/total_loss‚56.21347235760439 _step‚221 .loss/FBt1/detection_loss‚4.609753052393596 /loss/FBt2/detection_loss‚0.7347681431230553 1loss/SRqqt2/assignment_loss‚2.315272838498155 /loss/FBt3/assignment_loss‚0.607723370282786 -loss/FRt4/detection_loss‚2.32286078354408 /loss/FBt3/detection_loss‚2.4469729087182452 trainer/global_step‚2699 /loss/FRt1/assignment_loss‚4.167041106101794 1loss/SRqqt1/assignment_loss‚1.793498603105545 .loss/FRt2/detection_loss‚2.501527408533024 .loss/FBt4/detection_loss‚3.543661469221115 0loss/FRt2/assignment_loss‚3.5410891337828203 2loss/SRqqt4/assignment_loss‚2.2056028046602503 1loss/FBt4/assignment_loss‚0.40525355184799994 0loss/SRqqt2/detection_loss‚2.396424170051302 1loss/SRqqt3/detection_loss‚2.4446883056844984 _runtime‚2459.124859048 /loss/FBt1/assignment_loss‚0.287917032110272 1loss/FBt2/assignment_loss‚0.34843825126426964 % _timestamp‚1.7884427420776625e+09ÝŒÆ5žÕ‚(:“ öðåÔý£ÒÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kdìœQÖ‚(:Æ öðåÔº±æÉ disk.sda2.in‚0& disk.nvme0n1.out‚11037.81298828125 disk.drbd2265.in‚0 cpu‚25.2870985747057 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125# disk.dm-0.out‚11037.81298828125 proc.cpu.threads‚17 disk.sda3.out‚ 89.98828125 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.649413108825684 network.sent‚ 5.727093e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11037.81298828125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 89.98828125( proc.memory.availableMB‚ 569.59765625$ disk./.usageGB‚293.1285209655762 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2727.83984375 network.recv‚893519* disk./.usagePercent‚19.672100993210925 disk.rbd1.out‚ 16.638671875 disk.rbd0.in‚0 disk.sda.out‚ 89.98828125 disk.nvme0n1.in‚0.078125$ memory_percent‚96.523451805114756Dɓׂ(:ˆ …ñåÔ¹“²Ç gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.337' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é¨nSØ‚(:È …ñåÔË­ÂÊ disk.rbd1.out‚ 16.701171875 disk.sda.out‚ 91.55078125 disk.sda3.out‚ 91.55078125 disk.dm-1.in‚0 disk.dm-1.out‚ 91.55078125 disk.dm-0.in‚0.078125$ memory_percent‚96.52426242828369 cpu‚25.285661786410046 network.sent‚ 5.83539e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚11084.14111328125 proc.cpu.threads‚17& disk.nvme0n1.out‚11084.14111328125 disk.drbd2265.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 569.46484375# proc.memory.rssMB‚ 2727.83984375 network.recv‚898757 disk.nvme0c0n1.in‚0.078125* proc.memory.percent‚16.649413108825684* disk./.usagePercent‚19.672940441915703 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0% disk./.usageGB‚293.14102935791016 disk.nvme0n1.in‚0.078125# disk.dm-0.out‚11084.14111328125 disk.drbd2266.out‚0 disk.rbd1.in‚0¨OÓق(:ˆ ”ñåÔ¹»µÇ gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£9á7BÚ‚(:· ”ñåÔöÏÃÊ* proc.memory.percent‚16.649413108825684# disk.dm-0.out‚11138.09326171875 disk.rbd1.out‚ 16.701171875 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 proc.cpu.threads‚17 disk.dm-1.out‚92.25 disk.rbd0.in‚0 disk.sda.in‚0$ memory_percent‚96.52426242828369 network.sent‚ 5.841324e+06% disk./.usageGB‚293.15488052368164 disk.sda3.out‚92.25 disk.drbd1716.in‚0& disk.nvme0n1.out‚11138.09326171875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 cpu‚25.284256622490123 network.recv‚903344 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.sda.out‚92.25( proc.memory.availableMB‚ 569.46484375* disk./.usagePercent‚19.673870005272526 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚11138.09326171875# proc.memory.rssMB‚ 2727.83984375 disk.rbd0.out‚0Á|R2Û- _wandbruntime‚2503  _runtime‚2503¯Ò”ˆÈÜ trainer/global_step‚2749 $ _timestamp‚1.788442786802066e+09 _runtime‚2503.849330254 _step‚222 ' lr-AdamW/pg1‚0.00014655135435390378 ' lr-AdamW/pg2‚0.00014655135435390378 ÉÆáÍÝÇ trainer/global_step‚2749 $ _timestamp‚1.788442786802066e+09 _runtime‚2503.849330254 _step‚222 ' lr-AdamW/pg1‚0.00014655135435390378 ' lr-AdamW/pg2‚0.00014655135435390378ÞX×2Þ- _wandbruntime‚2503  _runtime‚2503lÿü9ßßëO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108350034 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2590908541805 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213956378251264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43001691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14462093 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191746534283765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194442632719872 _step‚223 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191747332523885 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189156423982080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194442632719872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2600858332672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59852103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28539601 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2600075207680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ÑSÍ48521042 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108373145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 _runtime‚2503.850263225 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197043491052544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197042690757120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43002226 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2735576682496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59828994 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59828994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48521042 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48521040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197042690757120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2600075207680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14462625=;óuö'àðO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2600858332672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43001691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14462093 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191746534283765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194442632719872 _step‚223 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191747332523885 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189156423982080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194442632719872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2600858332672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59852103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28539601 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2600075207680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48521042 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108373145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 _runtime‚2503.850263225 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197043491052544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197042690757120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43002226 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2735576682496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59828994 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59828994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48521042 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48521040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197042690757120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2600075207680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14462625 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108350034 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2590908541805 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213956378251264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28549590 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2601002693120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884427876766121e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197112249816064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59873463 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43016731 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214032060441600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48538080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214032035668992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194511264293376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194511264293376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194511247122944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2736745368576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59850328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108411545 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48538080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2749 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194511247122944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108411545 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189223224731648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136Õ-Χ!ãûM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14467672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216768586903040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197113050124800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2601785831424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14467141 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2591051297449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _step‚224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197113050124800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59850328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2736551234048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108411545 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189223224731648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48538082 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48538082 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28549590 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2601002693120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884427876766121e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197112249816064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59873463 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43016731 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214032060441600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48538080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214032035668992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194511264293376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194511264293376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194511247122944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2736745368576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59850328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108411545 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48538080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2749 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194511247122944 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200520658118144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49373042 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217740487768064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197874211395072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29039198 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200521458413568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2784605795840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788442832514742e+09 _runtime‚2549.562214086 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195133635141573 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636338752453 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197874211395072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60899046 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60899046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60922155 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43753174 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192497296389120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200520658118144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220524897712128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49373042 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220525093563904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195851776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195134433381693 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200521458413568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110295197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14713976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2646463893504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647247018496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110272086 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2646463893504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14714499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2784434716672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49373040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 171079168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2637119953213 trainer/global_step‚2799µv\Îö'òðO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200520658118144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220524897712128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49373042 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220525093563904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195851776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195134433381693 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200521458413568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110295197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14713976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2646463893504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647247018496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110272086 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2646463893504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14714499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2784434716672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49373040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 171079168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2637119953213 trainer/global_step‚2799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200520658118144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49373042 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217740487768064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197874211395072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29039198 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200521458413568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2784605795840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788442832514742e+09 _runtime‚2549.562214086 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195133635141573 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636338752453 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197874211395072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60899046 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60899046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60922155 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43753174 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192497296389120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010ãøà;|žó‚(:“ ÐñåÔÚðÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.905' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.380999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï\°·Oô‚(:Ä ÐñåÔÒ¸´Ê disk.sda.out‚ 94.390625 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125( proc.memory.availableMB‚ 570.08984375% disk./.usageGB‚293.20964431762695 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.52044773101807* proc.memory.percent‚16.649413108825684* disk./.usagePercent‚19.677545249434083 disk.dm-1.in‚0 network.recv‚921967 disk.dm-1.out‚ 94.390625 disk.drbd2265.in‚0 disk.rbd1.out‚ 17.3662109375 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme0c0n1.out‚11340.39111328125 disk.sda3.out‚ 94.390625& disk.nvme0n1.out‚11340.39111328125 disk.dm-0.in‚0.078125# proc.memory.rssMB‚ 2727.83984375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 5.967989e+06 disk.rbd0.out‚0 disk.drbd1716.out‚0# disk.dm-0.out‚11340.39111328125 cpu‚25.278705804483206N“SJ2õ- _wandbruntime‚2550  _runtime‚25504OýööM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60920380 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14719399 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110333597 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚200591017485824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200590217177088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStats\|Xo!Monitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217816169958400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110333597 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192564131217408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60943515 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49390082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49390080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60920380 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43768592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49390080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2785415752192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2638043909825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2637262708857 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60943515 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192564114178048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183648768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2647391378944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197942842968576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚2550.419981364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197942825798144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29049190 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43768078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29049193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2648174517248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884428333721244e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217816169958400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110333597 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192564131217408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60943515 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49390082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208421376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49390080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60920380 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43768592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49390080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2785415752192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2638043909825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2637262708857 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60943515 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192564114178048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183648768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2647391378944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197942842968576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚2550.419981364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197942825798144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29049190 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43768078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29049193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2648174517248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884428333721244e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 P2„- _wandbruntime‚2595  _runtime‚2595Þ–ã ¦…ëO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14965476 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201305772899840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224357967887872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2683331364621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2693635704320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224358176322560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44503735 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50225042 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29538801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221524597284864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203999425774592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2833395375616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50225040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195838185835520 _runtime‚2595.147944752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195838202874880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50225040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2693635704320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198520735999381 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201305790070272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2682550163861 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203999425774592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ZLDeviceStatsMonitor.on_train_batch_sta%Å hKrt/inactive_split_bytes.large_pool.peak‚ 172325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 $ _timestamp‚1.788442878100415e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50225042 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61992207 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61969098 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692852579328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61992207 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692852579328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29538798 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201305772899840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2849 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203998625479168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201305790070272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201305772899840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224357967887872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2683331364621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2693635704320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224358176322560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44503735 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50225042 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29538801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221524597284864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203999425774592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2833395375616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50225040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195838185835520 _runtime‚2595.147944752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195838202874880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50225040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2693635704320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198520735999381 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201305790070272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2682550163861 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203999425774592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 $ _timestamp‚1.788442878100415e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50225042 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61992207 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61969098 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692852579328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61992207 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692852579328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29538798 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201305772899840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2849 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203998625479168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201305790070272 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61990432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50242080 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112255649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224434844345856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2683474120265 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198589275985041 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50242082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2693780064768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112255649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14969891 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198588477744713 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112232512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204068984846848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2694563203072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204068184538112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44518681 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62013567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29548790 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201374421643776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62013567 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112255649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224434844345856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2683474120265 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198589275985041 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50242082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2693780064768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112255649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14969891 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198588477744713 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112232512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204068984846848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2694563203072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204068184538112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44518681 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62013567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29548790 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201374421643776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62013567 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512èi/<2Š- _wandbruntime‚2596  _runtime‚2596…§dÓY‹Ó _runtime‚2596.026477741 .loss/FRt2/detection_loss‚1.783943932275383 0loss/FRt2/assignment_loss‚3.6493924102004693 1loss/SRqqt4/assignment_loss‚2.090837841257453 0loss/SRqqt3/detection_loss‚2.320255762491471 /loss/FBt1/detection_loss‚3.9788992404937744 _step‚233 /loss/FRt3/assignment_loss‚5.101634687498996 1loss/FBt4/assignment_loss‚0.43839829742792064 % _timestamp‚1.7884428789793456e+09 0loss/FBt3/assignment_loss‚0.6307505983157569 0loss/FBt1/assignment_loss‚0.3891055832306544 1loss/FBt2/assignment_loss‚0.28627128447581074 /loss/FRt3/detection_loss‚2.7550592171518424 /loss/FRt4/detection_loss‚2.0030445856234382 1loss/SRqqt1/detection_loss‚2.3333660436839594 /loss/FBt4/detection_loss‚2.9803541362285615 %loss/total_loss‚54.55437201933954 epoch‚2 /loss/FRt1/assignment_loss‚4.355013368030389 1loss/SRqqt1/assignment_loss‚2.139084078916689 trainer/global_step‚2849 1loss/SRqqt4/detection_loss‚1.2673882424086331 /loss/FBt2/detection_loss‚0.7638591132292504 /loss/FRt1/detection_loss‚2.4591432996094227 /loss/FRt4/assignment_loss‚3.276885252150278 2loss/SRqqt2/assignment_loss‚2.0554683089256285 1loss/SRqqt2/detection_loss‚2.6463821041584015 .loss/FBt3/detection_loss‚2.740140113979578 2loss/SRqqt3/assignment_loss‚2.1096945175757775¿DQ‰^ŒØ /loss/FRt3/detection_loss‚2.7550592171518424 /loss/FRt4/detection_loss‚2.0030445856234382 1loss/SRqqt1/detection_loss‚2.3333660436839594 /loss/FBt4/detection_loss‚2.9803541362285615 %loss/total_loss‚54.55437201933954 epoch‚2 /loss/FRt1/assignment_loss‚4.355013368030389 1loss/SRqqt1/assignment_loss‚2.139084078916689 trainer/global_step‚2849 1loss/SRqqt4/detection_loss‚1.2673882424086331 /loss/FBt2/detection_loss‚0.7638591132292504 /loss/FRt1/detection_loss‚2.4591432996094227 /loss/FRt4/assignment_loss‚3.276885252150278 2loss/SRqqt2/assignment_loss‚2.0554683089256285 1loss/SRqqt2/detection_loss‚2.6463821041584015 .loss/FBt3/detection_loss‚2.740140113979578 2loss/SRqqt3/assignment_loss‚2.1096945175757775 _runtime‚2596.026477741 .loss/FRt2/detection_loss‚1.783943932275383 0loss/FRt2/assignment_loss‚3.6493924102004693 1loss/SRqqt4/assignment_loss‚2.090837841257453 0loss/SRqqt3/detection_loss‚2.320255762491471 /loss/FBt1/detection_loss‚3.9788992404937744 _step‚233 /loss/FRt3/assignment_loss‚5.101634687498996 1loss/FBt4/assignment_loss‚0.43839829742792064 % _timestamp‚1.7884428789793456e+09 0loss/FBt3/assignment_loss‚0.6307505983157569 0loss/FBt1/assignment_loss‚0.3891055832306544 1loss/FBt2/assignment_loss‚0.28627128447581074éõã|“‚(:ˆ ŒòåÔã­¼Ç gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚189.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.9598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è8¦®IŽ‚(:¾ ŒòåÔ‹ÃßÊ disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda.out‚ 96.77734375 disk.dm-1.out‚ 96.77734375! disk.dm-0.out‚11518.029296875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚19.679292756862804 disk.drbd2265.out‚0$ disk.nvme0n1.out‚11518.029296875 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.649413108825684 cpu‚25.273398278913255 disk.drbd1716.out‚0$ disk./.usageGB‚293.2356834411621 disk.rbd1.out‚ 17.671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0$ memory_percent‚96.52266502380371# proc.memory.rssMB‚ 2727.83984375 network.sent‚ 6.175825e+06 disk.drbd2080.in‚0 disk.sda3.out‚ 96.77734375 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚11518.029296875 disk.dm-1.in‚0' proc.memory.availableMB‚ 569.7265625 network.recv‚943117&ºk&ž‚(:“ ›òåÔó‰¨Ç gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.504999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ PL‚(:Á ›òåÔÇŠ·Ê proc.cpu.threads‚17* disk./.usagePercent‚19.680116077092922 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 97.31640625 disk.dm-1.out‚ 97.31640625% disk.nvme0n1.out‚11564.0654296875" disk.dm-0.out‚11564.0654296875 disk.dm-0.in‚0.078125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚11564.0654296875 disk.sda.out‚ 97.31640625 disk.sda3.in‚0& proc.memory.availableMB‚ 569.453125 network.recv‚949280 disk.drbd2266.out‚0 disk.rbd0.out‚0 network.sent‚ 6.284973e+06 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.sda1.in‚0% disk./.usageGB‚293.24795150756836 disk.drbd2265.in‚0 disk.rbd1.out‚ 17.734375 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0$ memory_percent‚96.52433395385742 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2727.83984375* proc.memory.percent‚16.649413108825684 disk.drbd1716.out‚0 cpu‚25.272046641798763ô%ñž‘‚(:“ ªòåÔÏÉÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}/H§L’‚(:Á ªòåÔ‘“©Ë* proc.memory.percent‚16.649413108825684 cpu‚25.27087344023791( disk.nvme0c0n1.out‚11616.45947265625 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.rbd1.out‚ 17.859375$ disk./.usageGB‚293.2752456665039 disk.sda3.out‚ 97.8046875 disk.drbd2080.in‚0 network.sent‚ 6.291335e+06 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2727.83984375 disk.dm-1.in‚0 disk.drbd2266.out‚0$ memory_percent‚96.52435779571533 proc.cpu.threads‚17 network.recv‚953945 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 97.8046875& disk.nvme0n1.out‚11616.45947265625 disk.sda1.out‚0* disk./.usagePercent‚19.681947811000413# disk.dm-0.out‚11616.45947265625( proc.memory.availableMB‚ 569.44921875 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 97.8046875 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0ˆ,.Á2“- _wandbruntime‚2640  _runtime‚26403fo´Ç”Á & lr-AdamW/pg1‚0.0001450518679879908 & lr-AdamW/pg2‚0.0001450518679879908 trainer/global_step‚2899 % _timestamp‚1.7884429234257743e+09 _runtime‚2640.473013148 _step‚234ýMɦÌ•Æ _step‚234 & lr-AdamW/pg1‚0.0001450518679879908 & lr-AdamW/pg2‚0.0001450518679879908 trainer/global_step‚2899 % _timestamp‚1.7884429234257743e+09 _runtime‚2640.473013148êµ…2–- _wandbruntime‚2640  _runtime‚2640oÐNò'—ìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204737368745472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228191298863616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228191084137472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201907836857189 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204737351575040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207476592840192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199179075281920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207477393135616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201908635097309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45253809 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225308706801664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2740024390144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199179092321280 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63062259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225308682029056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114116190 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204737368745472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51077040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2739241265152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚2899 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63039150 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63039150 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2729542776029 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2728761575269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2882402108416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2640.473924528 _step‚235Ò@-µ÷'˜ñO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207476592840192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199179075281920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207477393135616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201908635097309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45253809 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225308706801664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2740024390144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199179092321280 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63062259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225308682029056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114116190 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204737368745472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51077040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2739241265152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚2899 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63039150 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63039150 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2729542776029 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2728761575269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2882402108416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2640.473924528 _step‚235 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204737368745472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228191298863616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214726144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228191084137472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201907836857189 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204737351575040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ë!h·2™- _wandbruntime‚2641  _runtime‚2641Ìf³D šöM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15219898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63060484 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204806000318976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45268288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30048393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201975578602521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2730466732641 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114154564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207546151899136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45268825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228267735745024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2883371525632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚236 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51094082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15220432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114177701 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51094082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51094080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63060484 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2641.335170305 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2740168750592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225384388992000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199245910110208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199245893070848 % _timestamp‚1.7884429242872624e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114177701 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114154564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201976376842849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51094080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30048390¨Ðä-'›ûM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15219898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63060484 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204806000318976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45268288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30048393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201975578602521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2730466732641 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114154564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207546151899136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45268825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228267735745024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2883371525632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚236 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51094082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15220432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114177701 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51094082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51094080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63060484 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2641.335170305 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2740168750592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225384388992000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199245910110208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199245893070848 % _timestamp‚1.7884429242872624e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114177701 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114154564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201976376842849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51094080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30048390콋Ãh2œ- _wandbruntime‚2641  _runtime‚2641'8QË % _timestamp‚1.7884429242892935e+09 _runtime‚2641.336441475 /loss/FRt1/assignment_loss‚4.249491528766911 /loss/FRt2/assignment_loss‚4.339507699012756 0loss/SRqqt1/detection_loss‚2.071117167792669 1loss/FBt2/assignment_loss‚0.22314815319990886 .loss/FRt1/detection_loss‚2.473851879195469 /loss/FRt4/detection_loss‚1.8890095635581372 trainer/global_step‚2899 2loss/SRqqt1/assignment_loss‚2.0590966038587615 /loss/FRt2/detection_loss‚2.2625058080810003 epoch‚2 _step‚237 -loss/FBt3/detection_loss‚3.21034969886144 /loss/FRt4/assignment_loss‚3.032221123353759 0loss/FBt3/assignment_loss‚0.4491056443753728 1loss/SRqqt2/detection_loss‚2.5096903267231854 1loss/SRqqt3/detection_loss‚2.6557476222515106 1loss/SRqqt4/detection_loss‚1.2492825562755268 /loss/FBt1/assignment_loss‚0.906166099011898 .loss/FBt1/detection_loss‚5.163718541463216 %loss/total_loss‚56.50410145886446 /loss/FRt3/assignment_loss‚4.101259486542808 2loss/SRqqt2/assignment_loss‚2.5416775711558084 /loss/SRqqt3/assignment_loss‚2.2896190546453 1loss/FBt4/assignment_loss‚0.28323900401592184 /loss/FBt2/detection_loss‚0.7661407610749302 2loss/SRqqt4/assignment_loss‚1.8196023004474464 -loss/FRt3/detection_loss‚2.77392135726081 .loss/FBt4/detection_loss‚3.184631907939911¨DÃVžÐ /loss/FRt4/assignment_loss‚3.032221123353759 0loss/FBt3/assignment_loss‚0.4491056443753728 1loss/SRqqt2/detection_loss‚2.5096903267231854 1loss/SRqqt3/detection_loss‚2.6557476222515106 1loss/SRqqt4/detection_loss‚1.2492825562755268 /loss/FBt1/assignment_loss‚0.906166099011898 .loss/FBt1/detection_loss‚5.163718541463216 %loss/total_loss‚56.50410145886446 /loss/FRt3/assignment_loss‚4.101259486542808 2loss/SRqqt2/assignment_loss‚2.5416775711558084 /loss/SRqqt3/assignment_loss‚2.2896190546453 1loss/FBt4/assignment_loss‚0.28323900401592184 /loss/FBt2/detection_loss‚0.7661407610749302 2loss/SRqqt4/assignment_loss‚1.8196023004474464 -loss/FRt3/detection_loss‚2.77392135726081 .loss/FBt4/detection_loss‚3.184631907939911 % _timestamp‚1.7884429242892935e+09 _runtime‚2641.336441475 /loss/FRt1/assignment_loss‚4.249491528766911 /loss/FRt2/assignment_loss‚4.339507699012756 0loss/SRqqt1/detection_loss‚2.071117167792669 1loss/FBt2/assignment_loss‚0.22314815319990886 .loss/FRt1/detection_loss‚2.473851879195469 /loss/FRt4/detection_loss‚1.8890095635581372 trainer/global_step‚2899 2loss/SRqqt1/assignment_loss‚2.0590966038587615 /loss/FRt2/detection_loss‚2.2625058080810003 epoch‚2 _step‚237 -loss/FBt3/detection_loss‚3.21034969886144íãâ9i“Ÿ‚(:ˆ ¹òåÔÉÁÙÇ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¹1K ‚(:À ¹òåÔþ‚óÉ disk.drbd1716.in‚0& disk.nvme0n1.out‚11660.74853515625 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.649413108825684 disk.drbd2266.out‚0 disk.rbd1.out‚ 17.921875# proc.memory.rssMB‚ 2727.83984375 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0& proc.memory.availableMB‚ 569.421875 disk.dm-1.out‚ 98.3046875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.out‚0* disk./.usagePercent‚19.682433713300398 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚11660.74853515625 disk.sda.out‚ 98.3046875 disk.sda3.out‚ 98.3046875 network.sent‚ 6.296517e+06 network.recv‚957247 disk.drbd2080.out‚0# memory_percent‚96.5245246887207 disk.dm-1.in‚0# disk.dm-0.out‚11660.74853515625 disk.drbd2265.out‚0 disk.sda2.in‚0 cpu‚25.269572210567482 proc.cpu.threads‚17% disk./.usageGB‚293.28248596191406Z :“¡‚(:ˆ ÈòåÔ•±áÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßçýrN¢‚(:à ÈòåÔ®­¢Ê( proc.memory.availableMB‚ 569.66015625# proc.memory.rssMB‚ 2727.83984375 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚11719.9130859375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 cpu‚25.268343648408774 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.in‚0.078125$ memory_percent‚96.52307033538818* proc.memory.percent‚16.649413108825684 disk.drbd2080.out‚0 disk.sda.out‚ 98.78515625 disk.sda3.out‚ 98.78515625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚11719.9130859375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 98.78515625* disk./.usagePercent‚19.683140806109968$ disk./.usageGB‚293.2930221557617 disk.sda2.out‚0 disk.rbd1.out‚ 18.1015625 disk.rbd0.out‚0 network.sent‚ 6.404841e+06 disk.sda.in‚0 network.recv‚962799 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚11719.9130859375 disk.drbd2266.out‚0¤K¯‘£‚(:† ×òåÔß»ÃÇ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÒëN¤‚(:à ×òåÔ«»®É disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2727.83984375$ disk./.usageGB‚293.3063316345215' disk.nvme0c0n1.out‚11759.7958984375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 99.37109375 disk.drbd2265.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.649413108825684 disk.drbd2266.in‚0* disk./.usagePercent‚19.684034016396936 disk.rbd0.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚11759.7958984375 disk.sda1.in‚0$ memory_percent‚96.52307033538818 network.sent‚ 6.410515e+06 network.recv‚966548 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2080.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 569.66015625 disk.sda2.out‚0 disk.rbd1.out‚ 18.1640625 proc.cpu.threads‚17 disk.drbd1716.in‚0" disk.dm-0.out‚11759.7958984375 disk.rbd1.in‚0 disk.sda.out‚ 99.37109375 disk.sda3.out‚ 99.37109375 disk.dm-1.in‚0 cpu‚25.267060775276835÷Æóæ2¥- _wandbruntime‚2685  _runtime‚2685c®RäǦÁ & lr-AdamW/pg1‚0.0001445235418759071 & lr-AdamW/pg2‚0.0001445235418759071 trainer/global_step‚2949 % _timestamp‚1.7884429687876182e+09 _runtime‚2685.834863987 _step‚238ýºHóÌ§Æ & lr-AdamW/pg1‚0.0001445235418759071 & lr-AdamW/pg2‚0.0001445235418759071 trainer/global_step‚2949 % _timestamp‚1.7884429687876182e+09 _runtime‚2685.834863987 _step‚238îu2¿˜2¨- _wandbruntime‚2685  _runtime‚2685*9Ÿi©ìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210954560201216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64132311 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64109202 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15464433 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205294937714997 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2774972986677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚208168947420672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931392492544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208168930250240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232023996182016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232024208811008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202519964728320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30538001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30537998 % _timestamp‚1.7884429687883406e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210955360496640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116038242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51929040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2785629950976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15463914 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46001912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202519981767680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2785629950976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210954560201216¤l‘@÷'ªñO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187856384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2775754187437 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208168947420672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210955360496640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116061353 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚239 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64132311 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51929040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229092816318464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931204636160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208168930250240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2685.835802529 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64109202 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2786413075968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46002434 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51929042 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2786413075968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51929042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229092791545856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2785629950976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15463914 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46001912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202519981767680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2785629950976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210954560201216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210954560201216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64132311 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64109202 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15464433 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205294937714997 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2774972986677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚208168947420672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931392492544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208168930250240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232023996182016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232024208811008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202519964728320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30538001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30537998 % _timestamp‚1.7884429687883406e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210955360496640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116038242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51929040 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2786557436416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30547990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2775896943081 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208237578994176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51946080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116076616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51946080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884429696342998e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205363477700657 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64153671 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2776678144049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116099753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46016919 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚ 2686.68209764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211024119260160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208237561823744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229168498508800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202586799556608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232100669662208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚240 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116099753 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932381671936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211024919568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2786557436416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64130536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229168473736192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2787340574720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64153671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30547993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208237561823744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51946082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232100880180736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202586782517248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51946082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46017445bäD‚'­úM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚208237578994176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116099753 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932381671936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211024919568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2786557436416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64130536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229168473736192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2787340574720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64153671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30547993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208237561823744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51946082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232100880180736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202586782517248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51946082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46017445 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205362679460329 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15469452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2932195926016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2786557436416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30547990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2775896943081 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208237578994176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51946080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116076616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51946080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 % _timestamp‚1.7884429696342998e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205363477700657 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64153671 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2776678144049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116099753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46016919 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚ 2686.68209764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211024119260160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208237561823744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229168498508800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202586799556608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232100669662208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚240 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211600526095872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2821965598845 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214433327857664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832018636800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208434688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208682836812925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235857045100544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2980119265280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚2730.579218668 trainer/global_step‚2999 % _timestamp‚1.7884430135317938e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52781042 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211600508925440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52781040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117983405 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65202363 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235856836665856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208682038572805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31037598 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2979935603200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117960294 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214433327857664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214432527562240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205860854174720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211600508925440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2832801761792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31037601 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2832801761792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211600526095872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46752621 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117960294 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214433327857664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214432527562240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205860854174720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211600508925440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2832801761792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31037601 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2832801761792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211600526095872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46752621 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211600526095872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2821965598845 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214433327857664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832018636800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208434688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208682836812925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235857045100544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2980119265280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚2730.579218668 trainer/global_step‚2999 % _timestamp‚1.7884430135317938e+09ó}Ù=2½- _wandbruntime‚2731  _runtime‚2731>Ó͵2¾õM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117998668 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214502886929920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2999 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211669157669376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118021805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208750578558465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2832946122240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31047590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46767512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46766980 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211669140498944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2832946122240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118021805 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52798080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214502086621184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2833729260544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52798082 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232952608025600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚244 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52798082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52798080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232952583252992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 % _timestamp‚1.7884430143801484e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65223723 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214502886929920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65223723 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 2731.42780902bZy}'¿úM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚244 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52798082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52798080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232952583252992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 % _timestamp‚1.7884430143801484e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65223723 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214502886929920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65223723 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 2731.42780902 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117998668 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214502886929920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2999 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211669157669376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118021805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208750578558465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2832946122240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31047590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46767512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46766980 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211669140498944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2832946122240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118021805 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52798080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚214502086621184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2833729260544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52798082 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232952608025600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521ôéH>2À- _wandbruntime‚2731  _runtime‚2731›žUÁÏ % _timestamp‚1.7884430143818076e+09 0loss/SRqqt2/detection_loss‚2.491647472339017 1loss/SRqqt3/detection_loss‚2.3766233415314644 1loss/SRqqt4/detection_loss‚1.2415069756554622 .loss/FBt2/detection_loss‚0.672075883122241 epoch‚2 .loss/FBt4/detection_loss‚4.513556857903798 0loss/FRt2/assignment_loss‚3.3491190701723097 1loss/SRqqt2/assignment_loss‚2.479416725890977 /loss/FBt3/assignment_loss‚0.637033338700978 &loss/total_loss‚56.236094186527986 trainer/global_step‚2999 2loss/SRqqt1/assignment_loss‚1.8134041219949721 2loss/SRqqt4/assignment_loss‚2.0449267056058433 0loss/FBt4/assignment_loss‚0.5908326493390632 .loss/FRt1/detection_loss‚2.884066025953035 /loss/FRt3/detection_loss‚2.9999755116609426 .loss/FRt2/detection_loss‚2.296737379580736 /loss/FBt1/detection_loss‚3.7773910462856293 /loss/FBt3/detection_loss‚2.7877488548939047 _step‚245 /loss/FRt1/assignment_loss‚4.006988493171898 /loss/FRt3/assignment_loss‚4.172738698812632 0loss/FRt4/assignment_loss‚3.0201705725569474 /loss/FRt4/detection_loss‚1.9911061368490521 _runtime‚2731.428928093 2loss/SRqqt3/assignment_loss‚2.4230623507138453 0loss/FBt1/assignment_loss‚0.7984597645699978 /loss/SRqqt1/detection_loss‚2.54849950330598 0loss/FBt2/assignment_loss‚0.3190067059172582nÞ(ZÂÔ 1loss/SRqqt2/assignment_loss‚2.479416725890977 /loss/FBt3/assignment_loss‚0.637033338700978 &loss/total_loss‚56.236094186527986 trainer/global_step‚2999 2loss/SRqqt1/assignment_loss‚1.8134041219949721 2loss/SRqqt4/assignment_loss‚2.0449267056058433 0loss/FBt4/assignment_loss‚0.5908326493390632 .loss/FRt1/detection_loss‚2.884066025953035 /loss/FRt3/detection_loss‚2.9999755116609426 .loss/FRt2/detection_loss‚2.296737379580736 /loss/FBt1/detection_loss‚3.7773910462856293 /loss/FBt3/detection_loss‚2.7877488548939047 _step‚245 /loss/FRt1/assignment_loss‚4.006988493171898 /loss/FRt3/assignment_loss‚4.172738698812632 0loss/FRt4/assignment_loss‚3.0201705725569474 /loss/FRt4/detection_loss‚1.9911061368490521 _runtime‚2731.428928093 2loss/SRqqt3/assignment_loss‚2.4230623507138453 0loss/FBt1/assignment_loss‚0.7984597645699978 /loss/SRqqt1/detection_loss‚2.54849950330598 0loss/FBt2/assignment_loss‚0.3190067059172582 % _timestamp‚1.7884430143818076e+09 0loss/SRqqt2/detection_loss‚2.491647472339017 1loss/SRqqt3/detection_loss‚2.3766233415314644 1loss/SRqqt4/detection_loss‚1.2415069756554622 .loss/FBt2/detection_loss‚0.672075883122241 epoch‚2 .loss/FBt4/detection_loss‚4.513556857903798 0loss/FRt2/assignment_loss‚3.3491190701723097õ”A¼V“Â(:ˆ “óåÔ´ÄòÈ gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÂ\=AÄ‚(:¶ “óåÔ‚§Ê disk.rbd0.out‚0 disk.drbd1716.in‚0 cpu‚25.26243323112583* disk./.usagePercent‚19.683741655792833 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚101.4375 disk.dm-0.in‚0.078125' proc.memory.availableMB‚ 569.5546875# proc.memory.rssMB‚ 2727.83984375 network.sent‚ 6.538036e+06 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.dm-1.out‚101.4375* proc.memory.percent‚16.649413108825684% disk./.usageGB‚293.30197525024414 disk.sda2.in‚0 disk.dm-0.out‚11948.44140625 disk.rbd1.out‚ 18.3837890625 disk.sda1.out‚0% disk.nvme0c0n1.out‚11948.44140625# disk.nvme0n1.out‚11948.44140625$ memory_percent‚96.52371406555176 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚101.4375 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 network.recv‚986586 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚00©t)“Å‚(:ˆ ¢óåÔÄòÜÇ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.307' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÞ*‰OÆ‚(:Ä ¢óåÔè¦ÄÉ disk.drbd2080.out‚0 disk.sda3.out‚ 101.99609375( proc.memory.availableMB‚ 569.30078125# proc.memory.rssMB‚ 2727.83984375* proc.memory.percent‚16.649413108825684 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0$ memory_percent‚96.52526378631592 proc.cpu.threads‚17 network.recv‚993651* disk./.usagePercent‚19.685299717646217 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚12000.98046875 disk.dm-0.out‚12000.98046875# disk.nvme0n1.out‚12000.98046875 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 101.99609375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 18.5087890625% disk./.usageGB‚293.32519149780273 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 101.99609375 disk.sda2.out‚0 cpu‚25.261304089899973 network.sent‚ 6.647264e+06ÚÍW 2Ç- _wandbruntime‚2774  _runtime‚2774GÜÃâÉÈà ' lr-AdamW/pg2‚0.00014342479981475486 trainer/global_step‚3049 % _timestamp‚1.7884430577686474e+09 _runtime‚2774.815874154 _step‚246 ' lr-AdamW/pg1‚0.00014342479981475486Žé|ÎÉÈ _step‚246 ' lr-AdamW/pg1‚0.00014342479981475486 ' lr-AdamW/pg2‚0.00014342479981475486 trainer/global_step‚3049 % _timestamp‚1.7884430577686474e+09 _runtime‚2774.815874154öX9w2Ê- _wandbruntime‚2774  _runtime‚2774¦{L6ò'ËìO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66272415 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3028916926464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2878407322624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209201743621120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217910494923264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15964057 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119882346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2878407322624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209201760660480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53633042 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2879190447616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028722778624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 trainer/global_step‚3049 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66272415 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119905457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15964574 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217911295218688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47501255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239689733358080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 _runtime‚2774.816825149 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239689952278528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47501775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236661035352064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884430577693677e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215032104771072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194147840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217910494923264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217911295218688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2879190447616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53633042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31537198 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53633040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215032104771072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66249306 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212069139430613 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215032087600640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BJÌñO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 _runtime‚2774.816825149 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.al ¥v­%l.current‚520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239689952278528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47501775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236661035352064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884430577693677e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215032104771072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194147840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217910494923264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217911295218688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2879190447616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53633042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31537198 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53633040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215032104771072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66249306 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212069139430613 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215032087600640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217910494923264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15964057 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119882346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2878407322624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209201760660480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53633042 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2879190447616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028722778624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 trainer/global_step‚3049 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66272415 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119905457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15964574 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217911295218688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47501255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239689733358080÷í8jÌÍ‚(:’ ±óåÔ·Ž¼Ç gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.97' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.193999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¸DE΂(:º ±óåÔÉ·£Ê# disk./.usageGB‚293.334716796875 disk.sda.out‚ 102.640625 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd1716.in‚0# disk.nvme0n1.out‚12042.75390625 disk.dm-0.out‚12042.75390625 disk.drbd2080.out‚0 disk.rbd1.in‚0 network.sent‚ 6.6526e+06 disk.rbd0.out‚0% disk.nvme0c0n1.out‚12042.75390625 disk.drbd2265.out‚0* proc.memory.percent‚16.649413108825684 cpu‚25.260158825589347 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 102.640625 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0* disk./.usagePercent‚19.685938968459215 disk.dm-1.out‚ 102.640625 disk.drbd2266.in‚0$ memory_percent‚96.52712345123291# proc.memory.rssMB‚ 2727.83984375 disk.rbd1.out‚ 18.6572265625( proc.memory.availableMB‚ 568.99609375 proc.cpu.threads‚17 network.recv‚997053 disk.sda2.out‚0«Ðº2Ï- _wandbruntime‚2775  _runtime‚2775‹Q<žü&ÐöM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31547190 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215100736344576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119943857 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217980854290944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3029688018944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66293775 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212136881175945 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119943857 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879334808064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212137679416273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217980854290944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217980053982208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66293775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879334808064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31547193 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3029884250624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 _runtime‚2775.660827288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2880117946368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215100736344576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236736717542400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66270640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239766601793024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2869100966865 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884430586130548e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15969554 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209268561410048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215100719174144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53650080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119920720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119920720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53650082 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2880117946368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47516747 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209268578449408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53650082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47516221 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15969031 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47516221 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15969031 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119943857 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217980854290944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3029688018944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66293775 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212136881175945 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119943857 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879334808064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212137679416273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217980854290944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217980053982208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66293775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879334808064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31547193 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3029884250624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 _runtime‚2775.660827288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2880117946368 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215100736344576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236736717542400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66270640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239766601793024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2869100966865 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884430586130548e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15969554 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209268561410048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215100719174144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53650080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119920720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119920720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53650082 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2880117946368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47516747 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209268578449408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53650082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221004288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376øëƒ#\2Ò- _wandbruntime‚2775  _runtime‚2775ÞÑ´VÓÐ .loss/FBt1/detection_loss‚3.678942839304606 /loss/FBt2/detection_loss‚0.7406341132782472 2loss/SRqqt1/assignment_loss‚2.0076220014265607 0loss/FBt4/assignment_loss‚0.3517931817304145 .loss/FRt1/detection_loss‚2.434971418380737 /loss/FRt2/detection_loss‚2.0678475989235774 /loss/FRt4/detection_loss‚2.2092504913990316 %loss/total_loss‚55.50078664397471 0loss/FBt1/assignment_loss‚0.6710297749232268 0loss/FBt3/assignment_loss‚0.6770672391761433 .loss/FRt3/detection_loss‚2.769503426551819 0loss/SRqqt3/detection_loss‚2.699055051164968 /loss/FRt1/assignment_loss‚4.363391697406769 0loss/FRt2/assignment_loss‚3.8865230798721315 /loss/FRt3/assignment_loss‚4.742874349866595 1loss/SRqqt1/detection_loss‚2.4719551021144506 /loss/FBt3/detection_loss‚2.2512501694939355 _step‚249 0loss/FRt4/assignment_loss‚3.4501051833996406 /loss/FBt4/detection_loss‚3.9698910415172577 % _timestamp‚1.7884430586147795e+09 1loss/SRqqt3/assignment_loss‚1.906128974897521 1loss/SRqqt4/assignment_loss‚2.255499244370359 0loss/SRqqt2/detection_loss‚2.425824924751564 2loss/SRqqt2/assignment_loss‚1.8342123870496396 1loss/FBt2/assignment_loss‚0.27243550440892395 0loss/SRqqt4/detection_loss‚1.362977848566593 epoch‚2 trainer/global_step‚3049 _runtime‚2775.661907637Vï“—{ÔÕ 1loss/SRqqt3/assignment_loss‚1.906128974897521 1loss/SRqqt4/assignment_loss‚2.255499244370359 0loss/SRqqt2)}rÓà/detection_loss‚2.425824924751564 2loss/SRqqt2/assignment_loss‚1.8342123870496396 1loss/FBt2/assignment_loss‚0.27243550440892395 0loss/SRqqt4/detection_loss‚1.362977848566593 epoch‚2 trainer/global_step‚3049 _runtime‚2775.661907637 .loss/FBt1/detection_loss‚3.678942839304606 /loss/FBt2/detection_loss‚0.7406341132782472 2loss/SRqqt1/assignment_loss‚2.0076220014265607 0loss/FBt4/assignment_loss‚0.3517931817304145 .loss/FRt1/detection_loss‚2.434971418380737 /loss/FRt2/detection_loss‚2.0678475989235774 /loss/FRt4/detection_loss‚2.2092504913990316 %loss/total_loss‚55.50078664397471 0loss/FBt1/assignment_loss‚0.6710297749232268 0loss/FBt3/assignment_loss‚0.6770672391761433 .loss/FRt3/detection_loss‚2.769503426551819 0loss/SRqqt3/detection_loss‚2.699055051164968 /loss/FRt1/assignment_loss‚4.363391697406769 0loss/FRt2/assignment_loss‚3.8865230798721315 /loss/FRt3/assignment_loss‚4.742874349866595 1loss/SRqqt1/detection_loss‚2.4719551021144506 /loss/FBt3/detection_loss‚2.2512501694939355 _step‚249 0loss/FRt4/assignment_loss‚3.4501051833996406 /loss/FBt4/detection_loss‚3.9698910415172577 % _timestamp‚1.7884430586147795e+09ùÞPu“Õ‚(:ˆ ÀóåÔ’¤×Ç gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Æ]<Ö‚(:± ÀóåÔðç¿Ê network.recv‚ 1.003337e+06 disk.nvme0n1.in‚0.078125! disk.nvme0n1.out‚ 12095.796875 disk.dm-0.out‚ 12095.796875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.6572265625# disk.nvme0c0n1.out‚ 12095.796875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚103.25* proc.memory.percent‚16.649413108825684 proc.cpu.threads‚17 disk.drbd1716.out‚0( proc.memory.availableMB‚ 569.20703125 disk.rbd0.in‚0$ disk./.usageGB‚293.3486976623535 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚103.25 cpu‚25.259035139444457 network.sent‚ 6.750407e+06* disk./.usagePercent‚19.686877236072206 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚96.52583599090576 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚103.25# proc.memory.rssMB‚ 2727.83984375n®“ׂ(:ˆ ÏóåÔÂðÜÇ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÅžLØ‚(:Á ÏóåÔ¤êÂÉ disk.rbd0.in‚0 disk.dm-0.out‚ 12121.9453125# proc.memory.rssMB‚ 2727.83984375 network.recv‚ 1.00924e+06 disk.rbd1.out‚ 18.8994140625 disk.nvme0c0n1.in‚0.078125 disk.drbd2080.in‚0$ memory_percent‚96.52731418609619* proc.memory.percent‚16.649413108825684 disk.sda1.out‚0 disk.dm-1.out‚ 104.0078125" disk.nvme0n1.out‚ 12121.9453125( proc.memory.availableMB‚ 568.96484375 disk.rbd1.in‚0 disk.sda.out‚ 104.0078125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.25794561708589 disk.sda2.out‚0 disk.dm-0.in‚0.078125* disk./.usagePercent‚19.687436612534523$ disk.nvme0c0n1.out‚ 12121.9453125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚293.3570327758789 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 104.0078125 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 network.sent‚ 6.848647e+06||2Ù- _wandbruntime‚2819  _runtime‚2819ˆÝÅÚ¿ % lr-AdamW/pg1‚0.000142854612841885 % lr-AdamW/pg2‚0.000142854612841885 trainer/global_step‚3099 % _timestamp‚1.7884431024435718e+09 _runtime‚2819.490833071 _step‚250& ÊÛÄ % _timestamp‚1.7884431024435718e+09 _runtime‚2819.490833071 _step‚250 % lr-AdamW/pg1‚0.000142854612841885 % lr-AdamW/pg2‚0.000142854612841885 trainer/global_step‚3099úàŒ×Å2Ü- _wandbruntime‚2819  _runtime‚2819îéA[ñ'ÝëO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2924796008448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54485042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16214756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240445144868864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚3099 $ _timestamp‚1.788443102444307e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 _step‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67342467 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221388462284288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67342467 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243523063582720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67319358 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2913607220901 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218463683446272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121804398 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218463666275840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210531840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121827509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3077732954624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48252109 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48251554 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2924796008448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2925579133440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215457038528541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215456240288421 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54485040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243522853050880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67319358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221388462284288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2925579133440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218463683446272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2924796008448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2925579133440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215457038528541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215456240288421 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54485040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243522853050880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67319358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221388462284288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2925579133440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218463683446272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54485042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16214756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240445144868864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚3099 $ _timestamp‚1.788443102444307e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 _step‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67342467 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221388462284288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67342467 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243523063582720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67319358 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2913607220901û×’þ“ß‚(:ˆ ÞóåÔó §È gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p(X6[à‚(:Ð ÞóåÔêÔÒÉ# disk.dm-0.out‚12183.36865234375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.25684741330688 proc.cpu.threads‚17 network.sent‚ 6.855099e+06 network.recv‚ 1.014337e+06 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0$ disk./.usageGB‚293.3650093078613 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 19.2900390625' proc.memory.availableMB‚ 569.2109375* proc.memory.percent‚16.649413108825684 disk.sda.in‚0 disk.dm-1.out‚ 104.49609375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚96.52581214904785 disk.sda1.in‚0# proc.memory.rssMB‚ 2727.83984375 disk.sda.out‚ 104.49609375 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125& disk.nvme0n1.out‚12183.36865234375 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚12183.36865234375 disk.sda3.in‚0 disk.sda3.out‚ 104.49609375* disk./.usagePercent‚19.687971924288618 disk.dm-1.in‚0Ôžy2á- _wandbruntime‚2820  _runtime‚2820'IJ âöM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225198592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54502082 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221458021343232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218532315019776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240520827059200 trainer/global_step‚3099 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121865909 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚218532297849344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243599717634560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121865909 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16220291 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2925723493888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32046790 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16219746 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212609450856448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 _runtime‚2820.327961074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218532315019776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2926506632192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3078690149376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240520802286592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221458821651968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121842772 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54502082 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67363827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48266536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2914531177305 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121842772 J=DeviceStatsMonitor.on_train_batch_end;Ñ¡ýí/active.small_pool.freed‚67340692 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48267084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215524780274081 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32046793 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243599492435968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2915312378273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54502080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121865909 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16220291 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2925723493888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32046790 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16219746 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212609450856448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 _runtime‚2820.327961074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218532315019776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2926506632192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3078690149376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240520802286592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221458821651968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121842772 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54502082 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67363827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48266536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2914531177305 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121842772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67340692 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48267084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215524780274081 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32046793 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243599492435968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2915312378273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54502080 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225198592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54502082 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221458021343232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218532315019776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240520827059200 trainer/global_step‚3099 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121865909 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521üHÒÇ2ä- _wandbruntime‚2820  _runtime‚2820ÝPåQåË -loss/FRt4/detection_loss‚2.26079715820069 trainer/global_step‚3099 /loss/FRt2/detection_loss‚2.2399161242139645 1loss/SRqqt2/assignment_loss‚2.350667933194802 0loss/FBt2/assignment_loss‚0.2951373270165281 0loss/SRqqt4/detection_loss‚1.383382437314759 /loss/FBt2/detection_loss‚0.7629854343516591 1loss/SRqqt1/assignment_loss‚2.227852999470955 .loss/FBt1/detection_loss‚3.026746074358622 .loss/FBt4/detection_loss‚3.617162210600717 0loss/FBt4/assignment_loss‚0.4605602132121151 /loss/FRt2/assignment_loss‚4.045501941289658 /loss/FRt4/assignment_loss‚3.079213559627533 /loss/FBt3/detection_loss‚2.3285392274459205 1loss/SRqqt2/detection_loss‚2.0355293740486275 1loss/SRqqt3/detection_loss‚2.2808915093923225 _step‚253 /loss/FRt1/assignment_loss‚4.472067651294527 /loss/FRt3/assignment_loss‚4.610963457669968 1loss/SRqqt4/assignment_loss‚2.044063558920901 0loss/FBt1/assignment_loss‚0.5241097361625483 /loss/FRt3/detection_loss‚2.7025760351083217 0loss/SRqqt1/detection_loss‚2.168597800787105 _runtime‚2820.328981691 2loss/SRqqt3/assignment_loss‚2.2893669184965963 epoch‚2 $ _timestamp‚1.788443103281873e+09 0loss/FBt3/assignment_loss‚0.5421604054669539 .loss/FRt1/detection_loss‚2.688507011958531 &loss/total_loss‚54.437296099604325ù=µVæÐ trainer/global_step‚3099 /loss/FRt2/detection_loss‚2.2399161242139645 1loss/SRqqt2/assignment_loss‚2.350667933194802 0loss/FBt2/assignment_loss‚0.2951373270165281 0loss/SRqqt4/detection_loss‚1.383382437314759 /loss/FBt2/detection_loss‚0.7629854343516591 1loss/SRqqt1/assignment_loss‚2.227852999470955 .loss/FBt1/detection_loss‚3.026746074358622 .loss/FBt4/detection_loss‚3.617162210600717 0loss/FBt4/assignment_loss‚0.4605602132121151 /loss/FRt2/assignment_loss‚4.045501941289658 /loss/FRt4/assignment_loss‚3.079213559627533 /loss/FBt3/detection_loss‚2.3285392274459205 1loss/SRqqt2/detection_loss‚2.0355293740486275 1loss/SRqqt3/detection_loss‚2.2808915093923225 _step‚253 /loss/FRt1/assignment_loss‚4.472067651294527 /loss/FRt3/assignment_loss‚4.610963457669968 1loss/SRqqt4/assignment_loss‚2.044063558920901 0loss/FBt1/assignment_loss‚0.5241097361625483 /loss/FRt3/detection_loss‚2.7025760351083217 0loss/SRqqt1/detection_loss‚2.168597800787105 _runtime‚2820.328981691 2loss/SRqqt3/assignment_loss‚2.2893669184965963 epoch‚2 $ _timestamp‚1.788443103281873e+09 0loss/FBt3/assignment_loss‚0.5421604054669539 .loss/FRt1/detection_loss‚2.688507011958531 &loss/total_loss‚54.437296099604325 -loss/FRt4/detection_loss‚2.26079715820069ýk“ç‚(:ˆ íóåÔߢïÈ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06®«Å^è‚(:Ó íóåÔ¶ªöÊ disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚0.078125# disk.dm-0.out‚12228.45458984375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 105.29296875 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2727.83984375% disk./.usageGB‚293.37901306152344( disk.nvme0c0n1.out‚12228.45458984375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 cpu‚25.255739341466008 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 105.29296875( proc.memory.availableMB‚ 569.17578125* proc.memory.percent‚16.649413108825684 network.sent‚ 6.952116e+06* disk./.usagePercent‚19.688911727946813 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 19.6806640625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 105.29296875 disk.drbd1716.in‚0& disk.nvme0n1.out‚12228.45458984375 network.recv‚ 1.019976e+06$ memory_percent‚96.52602672576904 Óé“é‚(:ˆ üóåÔÀ‘ÅÇ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.123' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë`$ Vê‚(:Ë üóåÔÐÇ´É disk.rbd0.in‚0 disk.sda.out‚ 105.671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 19.6806640625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2727.83984375 disk.drbd1716.in‚0& disk.nvme0n1.out‚12274.67529296875 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125( proc.memory.availableMB‚ 569.41796875) disk./.usagePercent‚19.68908837314544 disk.rbd1.in‚0 disk.drbd2265.in‚0$ memory_percent‚96.52454853057861 network.recv‚ 1.026008e+06$ disk./.usageGB‚293.3816452026367 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12274.67529296875 disk.sda3.in‚0 disk.dm-1.out‚ 105.671875 disk.nvme0n1.in‚0.078125 cpu‚25.254607162477114 proc.cpu.threads‚17 network.sent‚ 7.050528e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 105.671875 disk.drbd1716.out‚0# disk.dm-0.out‚12274.67529296875* proc.memory.percent‚16.649413108825684 disk.sda1.out‚0bvj<2ë- _wandbruntime‚2863  _runtime‚2863åFF¦Éìà ' lr-AdamW/pg1‚0.00014227070604764887 ' lr-AdamW/pg2‚0.00014227070604764887 trainer/global_step‚3149 % _timestamp‚1.7884431465123665e+09 _runtime‚2863.559650894 _step‚254¦³;5ÎíÈ % _timestamp‚1.7884431465123665e+09 _runtime‚2863.559650894 _step‚254 ' lr-AdamW/pg1‚0.00014227070604764887 ' lr-AdamW/pg2‚0.00014227070604764887 trainer/global_step‚3149þþïÁã2î- _wandbruntime‚2863  _runtime‚2863ˆ{'àò'ïìO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55337042 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2971967819264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68412519 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49002767 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55337040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68389410 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247356052328448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16466369 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚3149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68412519 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32536398 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3126797942784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123726450 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224866429645312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68389410 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16466891 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215883539553280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2971967819264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2971184694272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218844139386349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49003292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221895262121472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244229229613056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2959818632309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224867229940736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224866429645312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3126605892096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123749561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55337040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244229254385664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2960599833069 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚255 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218843341146229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224867229940736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221895244951040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221895244951040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884431465130384e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224866429645312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3126605892096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123749561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55337040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244229254385664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2960599833069 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚255 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218843341146229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224867229940736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221895244951040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221895244951040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_tra‰ök«îin_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884431465130384e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68412519 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49002767 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55337040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68389410 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247356052328448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16466369 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 trainer/global_step‚3149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68412519 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32536398 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3126797942784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123726450 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224866429645312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68389410 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16466891 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215883539553280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2971967819264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2971184694272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218844139386349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49003292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221895262121472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244229229613056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2959818632309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224867229940736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489ÿV¬Ó¿2ñ- _wandbruntime‚2864  _runtime‚2864öbaü&òöM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224936789012992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2972895318016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68433879 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221963876524544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32546390 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215950357342208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123764824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215950340302848 _runtime‚2864.413991686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123787961 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221963893694976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68410744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247432710476288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244304911803392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16471343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55354080 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123764824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2972112179712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221963893694976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247432502054912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224936789012992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68433879 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123787961 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16471839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218911082891561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49018232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244304936576000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218911881131889 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55354080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49017733 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 10211875849»°'óûM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68410744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247432710476288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244304911803392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16471343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55354080 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123764824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2972112179712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221963893694976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247432502054912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224936789012992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68433879 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123787961 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16471839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218911082891561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49018232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244304936576000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218911881131889 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55354080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49017733 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224936789012992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2972895318016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68433879 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221963876524544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32546390 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215950357342208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123764824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215950340302848 _runtime‚2864.413991686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123787961 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221963893694976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967€Ê5x`2ô- _wandbruntime‚2864  _runtime‚2864@s­WõÑ 1loss/SRqqt2/assignment_loss‚2.074588212527727 1loss/SRqqt2/detection_loss‚2.7425876542141565 /loss/FBt4/detection_loss‚2.4562895397345224 %loss/total_loss‚54.34960783569413 2loss/SRqqt1/assignment_loss‚1.9700232168307175 2loss/SRqqt4/assignment_loss‚1.9313454058908281 0loss/FBt3/assignment_loss‚0.7736208811402321 1loss/SRqqt3/detection_loss‚2.3855829425156116 1loss/SRqqt4/detection_loss‚1.1483112086852392 0loss/FRt2/assignment_loss‚3.1496959805488585 0loss/FBt1/assignment_loss‚0.4845224107702961 0loss/FBt4/assignment_loss‚0.6876184114759477 /loss/FRt4/detection_loss‚1.9487517140805721 0loss/SRqqt1/detection_loss‚2.576896801188185 .loss/FBt3/detection_loss‚2.788463979959488 trainer/global_step‚3149 /loss/FRt1/detection_loss‚2.7070953164781844 /loss/FBt1/detection_loss‚3.7828715324401854 /loss/FRt3/assignment_loss‚4.785744650023324 0loss/FRt4/assignment_loss‚3.6509021259844303 0loss/SRqqt3/assignment_loss‚2.73567698802799 .loss/FRt3/detection_loss‚2.540010140055702 _runtime‚2864.415097713 /loss/FBt2/detection_loss‚0.6538162167095155 epoch‚2 /loss/FRt1/assignment_loss‚4.150875622885568 1loss/FBt2/assignment_loss‚0.24231830953816366 .loss/FRt2/detection_loss‚1.981998573988676 % _timestamp‚1.7884431473679373e+09 _step‚257“Oe©\öÖ /loss/FRt3/assignment_loss‚4.785744650023324 0loss/FRt4/assignment_loss‚3.6509021259844303 0loss/SRqqt3/assignment_loss‚2.73567698802799 .loss/FRt3/detection_loss‚2.540010140055702 _runtime‚2864.415097713 /loss/FBt2/detection_loss‚0.6538162167095155 epoch‚2 /loss/FRt1/assignment_loss‚4.150875622885568 1loss/FBt2/assignment_loss‚0.24231830953816366 .loss/FRt2/detection_loss‚1.981998573988676 % _timestamp‚1.7884431473679373e+09 _step‚257 1loss/SRqqt2/assignment_loss‚2.074588212527727 1loss/SRqqt2/detection_loss‚2.7425876542141565 /loss/FBt4/detection_loss‚2.4562895397345224 %loss/total_loss‚54.34960783569413 2loss/SRqqt1/assignment_loss‚1.9700232168307175 2loss/SRqqt4/assignment_loss‚1.9313454058908281 0loss/FBt3/assignment_loss‚0.7736208811402321 1loss/SRqqt3/detection_loss‚2.3855829425156116 1loss/SRqqt4/detection_loss‚1.1483112086852392 0loss/FRt2/assignment_loss‚3.1496959805488585 0loss/FBt1/assignment_loss‚0.4845224107702961 0loss/FBt4/assignment_loss‚0.6876184114759477 /loss/FRt4/detection_loss‚1.9487517140805721 0loss/SRqqt1/detection_loss‚2.576896801188185 .loss/FBt3/detection_loss‚2.788463979959488 trainer/global_step‚3149 /loss/FRt1/detection_loss‚2.7070953164781844 /loss/FBt1/detection_loss‚3.7828715324401854VÌ ~“÷‚(:ˆ ‹ôåÔòÚóÈ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Ü™ Vø‚(:Ë ‹ôåÔˆ™”Ë disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 106.3203125 disk.dm-1.out‚ 106.3203125 disk.drbd1716.in‚0$ memory_percent‚96.52829170227051* proc.memory.percent‚16.649413108825684 cpu‚25.25360032018717 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 106.3203125& disk.nvme0n1.out‚12318.75732421875 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚12318.75732421875 disk.sda.in‚0# proc.memory.rssMB‚ 2727.83984375 network.recv‚ 1.030126e+06) disk./.usagePercent‚19.68781499167013$ disk./.usageGB‚293.3626708984375# disk.dm-0.out‚12318.75732421875 disk.rbd1.out‚ 19.712890625 disk.sda1.in‚0 disk.nvme0n1.in‚0.078125' proc.memory.availableMB‚ 568.8046875 proc.cpu.threads‚17 network.sent‚ 7.056398e+06 disk.rbd1.in‚0¼²®=Ïù‚(:ˆ šôåÔ«áÎÇ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.846ÁÄéØÄ' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05÷0[6ú‚(:« šôåÔýÞåÉ disk.drbd1716.out‚0 disk.nvme0n1.out‚ 12371.90625 disk.sda1.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚0.078125# proc.memory.rssMB‚ 2727.83984375 network.sent‚ 7.165859e+06 network.recv‚ 1.037335e+06 disk.drbd1716.in‚0 disk.dm-0.out‚ 12371.90625 disk.sda2.in‚0 disk.sda3.out‚106.75* proc.memory.percent‚16.649413108825684 cpu‚25.252573968753946$ disk./.usageGB‚293.3901901245117 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚106.75 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0# memory_percent‚96.5285062789917 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.out‚ 19.837890625" disk.nvme0c0n1.out‚ 12371.90625 disk.sda.in‚0 disk.sda.out‚106.75 proc.cpu.threads‚17 disk.drbd2266.out‚0( proc.memory.availableMB‚ 568.76953125) disk./.usagePercent‚19.68966183002214¬¼©žû‚(:“ ©ôåÔ€¼ªÉ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¼ô¼Eü‚(:º ©ôåÔ¥á¹Ë disk.sda.out‚ 107.4609375 disk.drbd2266.out‚0 disk.drbd2265.in‚0$ memory_percent‚96.52566909790039* proc.memory.percent‚16.649413108825684 disk.dm-1.in‚0 disk.nvme0n1.out‚ 12414.03125 disk.dm-0.out‚ 12414.03125 cpu‚25.251418429682165" disk.nvme0c0n1.out‚ 12414.03125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 107.4609375 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.dm-1.out‚ 107.4609375 disk.rbd1.out‚ 19.900390625 proc.cpu.threads‚17 disk.rbd0.in‚0# proc.memory.rssMB‚ 2727.83984375 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚293.4040870666504 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 network.sent‚ 7.171251e+06* disk./.usagePercent‚19.690594465469378 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 569.234375 network.recv‚ 1.040912e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125ÇêÎ2ý- _wandbruntime‚2907  _runtime‚2907A]Èþ ' lr-AdamW/pg1‚0.00014167320111734726 ' lr-AdamW/pg2‚0.00014167320111734726 trainer/global_step‚3199 $ _timestamp‚1.788443190843703e+09 _runtime‚2907.890962912 _step‚258ÅBTªÍÿÇ trainer/global_step‚3199 $ _timestamp‚1.788443190843703e+09 _runtime‚2907.890962912 _step‚258 ' lr-AdamW/pg1‚0.00014167320111734726 ' lr-AdamW/pg2‚0.00014167320111734726‚¯ÆÊð2€- _wandbruntime‚2907  _runtime‚2907LKêÏò'ìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225326840796672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69459462 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219224428999680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 _runtime‚2907.891949283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228344397006336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125671613 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219224411960320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69482571 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3018356505088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248013363902464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69459462 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251189121463296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56189040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3175974384128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3006811244477 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3006030043717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225326840796672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33036001 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16716447 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251189338286592 trainer/global_step‚3199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125648502 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228344397006336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225326823626240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49752445 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222230442004037 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56189042 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3175782333440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69482571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33035998 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592µÇä=÷'‚ñO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56189042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16716986 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248013339129856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49752445 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222230442004037 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56189042 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3175782333440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69482571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33035998 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225326840796672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69459462 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219224428999680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 _runtime‚2907.891949283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228344397006336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125671613 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219224411960320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69482571 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3018356505088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248013363902464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69459462 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251189121463296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56189040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3175974384128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3006811244477 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3006030043717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225326840796672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33036001 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16716447 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251189338286592 trainer/global_step‚3199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125648502 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228344397006336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225326823626240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 % _timestamp‚1.7884431916706715e+09 _runtime‚2908.718577617 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33045993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3007735201089 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69480796 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69480796 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56206080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219291246788608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251265774021120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219291229749248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56206080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225395472370176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56206082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49767955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125686876 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56206082 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228414756374016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251265992928256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248089021320192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69503931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225395455199744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33045990 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248089046092800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49767414 BDeviceStatsMoniü01jAtor.on_train_batch_end/allocation.all.allocated‚ 125710013 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222298183749369 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3019284003840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69503931 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3019284003840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136ÿäæ'…ûM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248089021320192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69503931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225395455199744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33045990 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248089046092800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49767414 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125710013 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222298183749369 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3019284003840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69503931 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3019284003840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218907136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125710013 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194134528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 % _timestamp‚1.7884431916706715e+09 _runtime‚2908.718577617 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33045993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3007735201089 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69480796 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69480796 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56206080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219291246788608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251265774021120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219291229749248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56206080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225395472370176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56206082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49767955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125686876 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56206082 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228414756374016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251265992928256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808„÷E؉2†- _wandbruntime‚2908  _runtime‚2908ÒNUW‡Ñ 0loss/FBt4/assignment_loss‚0.2280418711182262 1loss/SRqqt1/detection_loss‚2.1046667604574134 .loss/FRt4/assignment_loss‚3.37040143032543 1loss/FBt2/assignment_loss‚0.23908145736369227 /loss/FRt1/detection_loss‚2.2758389841426503 /loss/FRt2/detection_loss‚1.8332379157113474 epoch‚2 2loss/SRqqt4/assignment_loss‚1.8610794025135564 1loss/SRqqt3/detection_loss‚2.5075788776079815 %loss/total_loss‚52.84679677672143 /loss/FRt1/assignment_loss‚4.265269548242743 0loss/FRt2/assignment_loss‚3.1386788487434387 2loss/SRqqt2/assignment_loss‚2.0845347086765935 0loss/FBt1/assignment_loss‚0.3171587262651883 /loss/FRt3/detection_loss‚2.8055506429156742 0loss/SRqqt4/detection_loss‚1.555122126344141 /loss/FBt3/detection_loss‚3.1037582953770957 $ _timestamp‚1.788443191672499e+09 .loss/FBt1/detection_loss‚3.497926926612854 /loss/FBt2/detection_loss‚0.7116351168176871 _step‚261 0loss/SRqqt2/detection_loss‚2.375874847173691 trainer/global_step‚3199 /loss/FRt3/assignment_loss‚4.167392340866295 2loss/SRqqt3/assignment_loss‚2.2940298239390056 .loss/FBt4/detection_loss‚3.598132950919015 _runtime‚2908.719624993 2loss/SRqqt1/assignment_loss‚2.0150219486760244 0loss/FBt3/assignment_loss‚0.6062400142327028 /loss/FRt4/detection_loss‚1.8905432116789895d¸Ì\ˆÖ 2loss/SRqqt4/assignment_loss‚1.8610794025135564 1loss/SRqqt3/detection_loss‚2.5075788776079815 %loss/total_loss‚52.84679677672143 /loss/FRt1/assignment_loss‚4.265269548242743 0loss/FRt2/assignment_loss‚3.1386788487434387 2loss/SRqqt2/assignment_loss‚2.0845347086765935 0loss/FBt1/assignment_loss‚0.3171587262651883 /loss/FRt3/detection_loss‚2.8055506429156742 0loss/SRqqt4/detection_loss‚1.555122126344141 /loss/FBt3/detection_loss‚3.1037582953770957 $ _timestamp‚1.788443191672499e+09 .loss/FBt1/detection_loss‚3.497926926612854 /loss/FBt2/detection_loss‚0.7116351168176871 _step‚261 0loss/SRqqt2/detection_loss‚2.375874847173691 trainer/global_step‚3199 /loss/FRt3/assignment_loss‚4.167392340866295 2loss/SRqqt3/assignment_loss‚2.2940298239390056 .loss/FBt4/detection_loss‚3.598132950919015 _runtime‚2908.719624993 2loss/SRqqt1/assignment_loss‚2.0150219486760244 0loss/FBt3/assignment_loss‚0.6062400142327028 /loss/FRt4/detection_loss‚1.8905432116789895 0loss/FBt4/assignment_loss‚0.2280418711182262 1loss/SRqqt1/detection_loss‚2.1046667604574134 .loss/FRt4/assignment_loss‚3.37040143032543 1loss/FBt2/assignment_loss‚0.23908145736369227 /loss/FRt1/detection_loss‚2.2758389841426503 /loss/FRt2/detection_loss‚1.8332379157113474 epoch‚2…8#P“‰‚(:ˆ ¸ôåÔæ×»Ç gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[öéGŠ‚(:¼ ¸ôåÔ¬ŸÁÊ disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 proc.cpu.threads‚17 network.sent‚ 7.177204e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125 disk.dm-0.out‚ 12469.359375 disk.sda.out‚ 107.796875$ memory_percent‚96.53191566467285 network.recv‚ 1.045438e+06% disk./.usageGB‚293.40421295166016 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 19.900390625 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 12469.359375! disk.nvme0n1.out‚ 12469.359375 disk.drbd2080.out‚0 disk.sda3.out‚ 107.796875 disk.dm-1.out‚ 107.796875 cpu‚25.250406453431115* disk./.usagePercent‚19.690602913718006 disk.sda1.in‚0 disk.drbd1716.out‚0' proc.memory.availableMB‚ 568.2109375# proc.memory.rssMB‚ 2727.83984375* proc.memory.percent‚16.649413108825684 disk.sda.in‚0ûwV6“‹‚(:ˆ ÇôåÔÜÒ¤Ç gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚191.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.2312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç.ì{HŒ‚(:½ ÇôåÔ׺éÉ network.recv‚ 1.052146e+06 disk.drbd1716.out‚0 disk.nvme0n1.out‚ 12509.4375 disk.sda1.out‚0 disk.sda.out‚ 108.50390625 disk.sda2.out‚0( proc.memory.availableMB‚ 568.42578125 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 108.50390625 disk.rbd0.in‚0* proc.memory.percent‚16.649413108825684 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 19.9326171875 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125# memory_percent‚96.5306043624878 disk.dm-1.out‚ 108.50390625 disk.dm-0.out‚ 12509.4375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.249399778570876 network.sent‚ 7.286589e+06* disk./.usagePercent‚19.692081357228243% disk./.usageGB‚293.42624282836914 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.out‚ 12509.4375# proc.memory.rssMB‚ 2727.83984375÷棉“‚(:ˆ ÖôåÔœß·Ç gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò„^)UŽ‚(:Ê ÖôåÔ ÒÂÊ network.sent‚ 7.292816e+06* disk./.usagePercent‚19.693131756141234 disk.dm-0.in‚0.078125 disk.sda1.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2727.83984375 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 108.9453125 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 20.0576171875 disk.sda.in‚0 disk.sda.out‚ 108.9453125& proc.memory.availableMB‚ 568.421875 network.recv‚ 1.056595e+06 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.649413108825684 cpu‚25.248379207371055" disk./.usageGB‚293.44189453125 disk.dm-1.out‚ 108.9453125# disk.dm-0.out‚12566.45068359375 disk.drbd2080.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚12566.45068359375( disk.nvme0c0n1.out‚12566.45068359375# memory_percent‚96.5306282043457Ê!SÖ2- _wandbruntime‚2951  _runtime‚2951°Ï¢=Éà ' lr-AdamW/pg2‚0.00014106222257011204 trainer/global_step‚3249 % _timestamp‚1.7884432346187384e+09 _runtime‚2951.665956971 _step‚262 ' lr-AdamW/pg1‚0.00014106222257011204tƒÖÀÎ‘È trainer/global_step‚3249 % _timestamp‚1.7884432346187384e+09 _runtime‚2951.665956971 _step‚262 ' lr-AdamW/pg1‚0.00014106222257011204 ' lr-AdamW/pg2‚0.00014106222257011204†5  02’- _wandbruntime‚2951  _runtime‚2951ï½1Zò'“ìO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231822364367360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228758419471872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3063962065920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255022357084672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3053022655885 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231823164662784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3063962065920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225617542861845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3224683226112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50502398 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16966800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228758402301440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57041042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33535601 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3064745190912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222565301406720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231822364367360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3064745190912 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57041040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50502938 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16967337 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225211904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251797473419264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70529514 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70552623 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70552623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251797448646656 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33535598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57041040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70529514 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228758402301440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255022131872768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0+·[(Ø ”ñO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3224883665408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222565318446080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3052241455125 trainer/global_step‚3249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57041042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231823164662784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200439296 % _timestamp‚1.7884432346194425e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127570554 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225618341101965 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57041040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50502938 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16967337 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225211904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251797473419264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70529514 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70552623 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70552623 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251797448646656 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33535598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LDDeviceStatsMoÛRŸ9nitor.on_train_batch_start/segment.large_pool.allocated‚521 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57041040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70529514 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228758402301440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255022131872768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231822364367360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228758419471872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3063962065920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255022357084672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3053022655885 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231823164662784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3063962065920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225617542861845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3224683226112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50502398 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16966800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228758402301440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57041042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33535601 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3064745190912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222565301406720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231822364367360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3064745190912 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70550848 _runtime‚2952.472810034 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70573983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16972318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 _step‚264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16971818 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251873155609600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231891923426304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3225680569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3064889551360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255098811406848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225685284607177 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57058080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127632065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3225862121472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3053165411529 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33545593 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231892723735040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228827033874944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222632136235008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 % _timestamp‚1.7884432354250832e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231892723735040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228827033874944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3065672689664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053946612497 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255099017731072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127632065 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206324224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50517911 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222632119195648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70550848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33545590 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127608928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57058080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3064889551360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50517408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228827051045376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228827051045376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251873130836992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181551616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968;6Ì'—ûM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3225680569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3064889551360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255098811406848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225685284607177 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57058080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127632065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3225862121472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3053165411529 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33545593 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231892723735040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228827033874944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222632136235008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 % _timestamp‚1.7884432354250832e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231892723735040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228827033874944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3065672689664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053946612497 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255099017731072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127632065 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206324224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50517911 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222632119195648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70550848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33545590 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127608928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57058080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3064889551360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50517408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228827051045376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228827051045376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251873130836992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181551616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70573983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127608928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70550848 _runtime‚2952.472810034 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70573983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16972318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 _step‚264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16971818 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251873155609600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231891923426304ˆ< &b2˜- _wandbruntime‚2952  _runtime‚2952~†:T™Î 0loss/FRt3/assignment_loss‚4.1446979488095925 /loss/FRt4/assignment_loss‚2.882470173140367 1loss/SRqqt1/assignment_loss‚2.088476801291108 /loss/FRt3/detection_loss‚2.9356071814413993 1loss/SRqqt4/detection_loss‚1.2076973729190372 _step‚265 /loss/FRt1/assignment_loss‚4.293434527942113 .loss/FRt4/detection_loss‚2.332741543340186 $loss/total_loss‚53.8146219709206 _runtime‚2952.473868231 .loss/FBt1/detection_loss‚4.372282162308693 /loss/FBt3/detection_loss‚2.6617868120471635 .loss/FBt4/detection_loss‚3.236618535085158 % _timestamp‚1.7884432354267304e+09 2loss/SRqqt3/assignment_loss‚2.0470438913024704 0loss/FBt1/assignment_loss‚0.3489419932302553 .loss/FRt1/detection_loss‚2.758520385197231 /loss/FRt2/detection_loss‚2.3114848643159256 0loss/SRqqt3/detection_loss‚2.523656705330158 epoch‚2 2loss/SRqqt2/assignment_loss‚1.7624562427401542 0loss/FBt2/assignment_loss‚0.2567640633658132 1loss/SRqqt1/detection_loss‚2.3228436052799224 /loss/FBt2/detection_loss‚0.7056967613776343 trainer/global_step‚3249 0loss/FRt2/assignment_loss‚2.9992864101361008 0loss/SRqqt2/detection_loss‚2.244983477592468 1loss/SRqqt4/assignment_loss‚2.131627580523491 0loss/FBt3/assignment_loss‚0.7052779671945851 0loss/FBt4/assignment_loss‚0.5402249650095737Œ! GYšÓ 1loss/SRqqt1/detection_loss‚2.3228436052799224 /loss/FBt2/detection_loss‚0.7056967613776343 trainer/global_step‚3249 0loss/FRt2/assignment_loss‚2.9992864101361008 0loss/SRqqt2/detection_loss‚2.244983477592468 1loss/SRqqt4/assignment_loss‚2.131627580523491 0loss/FBt3/assignment_loss‚0.7052779671945851 0loss/FBt4/assignment_loss‚0.5402249650095737 0loss/FRt3/assignment_loss‚4.1446979488095925 /loss/FRt4/assignment_loss‚2.882470173140367 1loss/SRqqt1/assignment_loss‚2.088476801291108 /loss/FRt3/detection_loss‚2.9356071814413993 1loss/SRqqt4/detection_loss‚1.2076973729190372 _step‚265 /loss/FRt1/assignment_loss‚4.293434527942113 .loss/FRt4/detection_loss‚2.332741543340186 $loss/total_loss‚53.8146219709206 _runtime‚2952.473868231 .loss/FBt1/detection_loss‚4.372282162308693 /loss/FBt3/detection_loss‚2.6617868120471635 .loss/FBt4/detection_loss‚3.236618535085158 % _timestamp‚1.7884432354267304e+09 2loss/SRqqt3/assignment_loss‚2.0470438913024704 0loss/FBt1/assignment_loss‚0.3489419932302553 .loss/FRt1/detection_loss‚2.758520385197231 /loss/FRt2/detection_loss‚2.3114848643159256 0loss/SRqqt3/detection_loss‚2.523656705330158 epoch‚2 2loss/SRqqt2/assignment_loss‚1.7624562427401542 0loss/FBt2/assignment_loss‚0.2567640633658132‰JçA“›‚(:ˆ åôåÔ¿ÉßÈ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èÿ÷^œ‚(:Ó åôåÔ€ÈäÉ% disk./.usageGB‚293.44939041137695 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚12613.66552734375 network.recv‚ 1.060098e+06 disk.nvme0c0n1.in‚0.078125 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 109.48828125 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125( disk.nvme0c0n1.out‚12613.66552734375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 20.2060546875 disk.sda1.in‚0* proc.memory.percent‚16.649413108825684 disk.drbd2080.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚12613.66552734375 disk.dm-0.in‚0.078125$ memory_percent‚96.53089046478271 network.sent‚ 7.298113e+06* disk./.usagePercent‚19.693634810946016 disk.sda3.out‚ 109.48828125 disk.dm-1.out‚ 109.48828125 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 568.37890625# proc.memory.rssMB‚ 2727.83984375 cpu‚25.247393771025585 disk.drbd2080.out‚0õ·±“‚(:ˆ ôôåÔš¯Ç gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.1442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦%TØZž‚(:Ï ôôåÔÕì¹É$ memory_percent‚96.53098583221436 network.sent‚ 7.407396e+06* disk./.usagePercent‚19.694901536225434 disk.sda3.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125( proc.memory.availableMB‚ 568.36328125 disk.dm-1.out‚ 109.85546875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚12672.3544921875# proc.memory.rssMB‚ 2727.83984375% disk./.usageGB‚293.46826553344727 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.649413108825684 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda.out‚ 109.85546875 network.recv‚ 1.067109e+06 disk.dm-0.in‚0.078125 cpu‚25.24641143048082 disk.sda2.out‚0 disk.sda3.out‚ 109.85546875 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 disk.rbd1.out‚ 20.2060546875 disk.rbd0.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚12672.3544921875" disk.dm-0.out‚12672.3544921875“Îî?¡Ÿ‚(:ˆ ƒõåÔ绣Ç gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gp:ÉÑtòu.0.temp‚37.0 gpu.0.powerWatts‚126.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GŸM(P ‚(:Å ƒõåÔÓ©Ê disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚12723.9189453125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 110.40625$ memory_percent‚96.53103351593018 cpu‚25.24541792513274 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 110.40625 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2727.83984375* proc.memory.percent‚16.649413108825684 disk.rbd1.out‚ 20.2685546875 disk.rbd0.out‚0 disk.dm-0.in‚0.078125( proc.memory.availableMB‚ 568.35546875 disk.drbd1716.out‚0 network.recv‚ 1.070097e+06) disk./.usagePercent‚19.69564191001446 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 110.40625 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚12723.9189453125" disk.dm-0.out‚12723.9189453125 network.sent‚ 7.412277e+06% disk./.usageGB‚293.47929763793945q¶Ú2¡- _wandbruntime‚2995  _runtime‚2995¥µ‚Ç¢Á _step‚266 & lr-AdamW/pg1‚0.0001404378977329567 & lr-AdamW/pg2‚0.0001404378977329567 trainer/global_step‚3299 % _timestamp‚1.7884432782828407e+09 _runtime‚2995.330009071AȘÑÌ£Æ trainer/global_step‚3299 % _timestamp‚1.7884432782828407e+09 _runtime‚2995.330009071 _step‚266 & lr-AdamW/pg1‚0.0001404378977329567 & lr-AdamW/pg2‚0.0001404378977329567Ššè2¤- _wandbruntime‚2995  _runtime‚2995ìÆ©‘ð'¥êO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71622675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255581582936064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235300331728384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129515717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235300331728384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232189980976640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3111133876736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17216644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17216115 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255581558163456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225906190853120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3099234067293 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273734367744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232189980976640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57893040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71622675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229005441959773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3111133876736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258855317303808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264ÓOcõ'¦ïO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57893040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71622675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229005441959773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3111133876736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258855317303808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71622675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255581582936064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235300331728384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129515717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235300331728384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232189980976640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3111133876736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17216644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17216115 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255581558163456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225906190853120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3099234067293 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273734367744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232189980976640‹žƒ$y2§- _wandbruntime‚2996  _runtime‚2996ôÁÖ°û&¨õM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788443279101846e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51266324 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71620900 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17221684 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232258612550144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129554117 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57910080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232258629720576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 _runtime‚2996.149700059 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129530980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258931973400576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57910080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3099376822937 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3112061375488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129554117 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51266877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3111278237184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71620900 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3100158023905 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255657240353792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129530980 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57910082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57910082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3111278237184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255657265126400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229073183705313 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788443279101846e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51266324 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71620900 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17221684 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232258612550144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129554117 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57910080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232258629720576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 _runtime‚2996.149700059 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129530980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258931973400576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57910080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3099376822937 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3112061375488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129554117 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51266877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3111278237184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71620900 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3100158023905 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255657240353792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129530980 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57910082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57910082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3111278237184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223101440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255657265126400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229073183705313 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238778299089408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235621559651840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262688318049792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262688099129344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259365667680256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884433214828184e+09 _runtime‚3038.530246418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72669618 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58745042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232392542817581 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3144664277941 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156739437568 _step‚271 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58745042 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72692727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235621559651840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235621576822272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52001556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229247097338880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131414658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72692727 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58745040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3322431449088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34534798 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17466230 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156739437568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17466755 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131437769 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3322625596928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52001028 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34534801 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3145445478701 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238778299089408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235621559651840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262688318049792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262688099129344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259365667680256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884433214828184e+09 _runtime‚3038.530246418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72669618 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58745042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232392542817581 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3144664277941 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156739437568 _step‚271 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58745042 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72692727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235621559651840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235621576822272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52001556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218920448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229247097338880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131414658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72692727 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58745040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3322431449088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34534798 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17466230 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156739437568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17466755 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131437769 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3322625596928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52001028 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34534801 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3145445478701 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229313898088448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131453032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72714087 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58762082 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232460284563121 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3158450061312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72714087 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52015967 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3323598631936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235690191225344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58762080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131476169 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131476169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235690208395776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212615680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238848658457088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259441374643200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235690191225344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72690952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3323410788864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229313915127808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34544793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262764760659456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157666923008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238848658457088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17471177 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58762082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235690208395776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚272 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010WÝqß'»ûM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58762082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235690208395776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚272 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259441349870592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58762080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229313898088448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131453032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72714087 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58762082 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232460284563121 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3158450061312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72714087 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52015967 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3323598631936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235690191225344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58762080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131476169 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131476169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235690208395776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212615680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238848658457088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259441374643200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235690191225344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72690952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3323410788864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229313915127808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34544793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262764760659456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157666923008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238848658457088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17471177 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210531840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242257066745856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73762779 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3203128123392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266521153329152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235779643675389 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133359821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263149777197056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35034398 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203911248384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3371561891328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232587969745920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59597040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73762779 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242256266450432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52750786 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚3399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266521363860992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 _runtime‚3081.649627993 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35034401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232587986785280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242257066745856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242256266450432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73739670 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59597042 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3371376132096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52750251 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133336710 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239053155497472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190875689349 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73739670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239053138327040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239053138327040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884433646021533e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3203128123392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17715853 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263149801969664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59597042 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521jœÇ÷'ÊñO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35034401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185759232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232587986785280 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242257066745856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242256266450432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73739670 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59597042 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3371376132096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52750251 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133336710 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239053155497472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190875689349 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73739670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239053138327040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239053138327040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884433646021533e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3203128123392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17715853 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263149801969664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59597042 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210531840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242257066745856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73762779 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3203128123392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266521153329152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235779643675389 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133359821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263149777197056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35034398 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203911248384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3371561891328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232587969745920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59597040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73762779 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242256266450432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52750786 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 trainer/global_step‚3399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266521363860992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 _runtime‚3081.649627993 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424“âôf 2Ë- _wandbruntime‚3082  _runtime‚3082òaœKü&ÌöM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235846587180601 _step‚276 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133398221 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242326625818112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59614082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73784139 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884433654117537e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133375084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59614080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52765649 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59614080 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239121787070976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3082.459509685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204838747136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266597800235520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59614082 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239121769900544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232654804574208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239121769900544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚3399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52765101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239121787070976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133398221 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133375084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263225484160000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3204055608832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239121769900544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚3399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52765101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239121787070976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_óRñz batch_end/allocation.all.allocated‚ 133398221 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133375084 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263225484160000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3204055608832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239121787070976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3082.459509685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204838747136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266597800235520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59614082 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239121769900544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232654804574208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245734233811456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35533998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17965962 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245735034106880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3420423748608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3237868301517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35534001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74809722 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266933911486464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239165946293077 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3237087100757 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242484734172672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242484717002240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60449042 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3250299934208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3249516809216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266933886713856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3250299934208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242484734172672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235928859192320 % _timestamp‚1.7884434080547488e+09ÒqùÜñO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245735034106880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3420423748608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3237868301517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35534001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74809722 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266933911486464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239165946293077 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3237087100757 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242484734172672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242484717002240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60449042 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3250299934208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3249516809216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266933886713856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3250299934208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor†ÖËþ.on_train_batch_start/active_bytes.large_pool.allocated‚242484734172672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235928859192320 % _timestamp‚1.7884434080547488e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135281873 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74832831 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚245734233811456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚239166744533197 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60449040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74832831 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚270354335235072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74809722 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242484717002240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚279 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245734233811456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35533998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17965962 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296—1ø2Ý- _wandbruntime‚3125  _runtime‚3125’!Üü&ÞöM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3421401506304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135297136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53514917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242553348575744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3251227432960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74831056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187843072 _runtime‚3125.904208217 _step‚280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3421213663232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235995694020608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3238011057161 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884434088563294e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74854191 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242553365746176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3251227432960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235995676981248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242553348575744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53515444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35543993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60466082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3250444294656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239234486278737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245804593179136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135320273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270430782567424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245804593179136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60466080 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚31578Fzð'ßûM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235995694020608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3238011057161 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 % _timestamp‚1.7884434088563294e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74854191 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242553365746176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3251227432960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235995676981248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242553348575744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53515444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35543993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60466082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3250444294656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239234486278737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245804593179136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135320273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270430782567424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245804593179136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60466080 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3421401506304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135297136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53514917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242553348575744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3251227432960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74831056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187843072 _runtime‚3125.904208217 _step‚280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021187584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3421213663232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3296688620032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36033598 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3284079712925 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274187063722496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249213001467904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54249440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 % _timestamp‚1.7884434517742248e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75902883 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3296688620032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245916295677440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75879774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18216380 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36033601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270717996230656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245916295677440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54249981 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18215842 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 trainer/global_step‚3499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249212201172480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61301040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61301042 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚283 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3469067491840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249212201172480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75902883 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239269765678080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270718021003264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239269748638720 _runtime‚3168.821721256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3469259542528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚242553845391005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3296688620032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36033598 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3284079712925 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274187063722496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249213001467904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54249440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 % _timestamp‚1.7884434517742248e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75902883 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3296688620032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245916295677440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75879774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18216380 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36033601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270717996230656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245916295677440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54249981 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18215842 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 trainer/global_step‚3499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249212201172480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61301040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216823296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61301042 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚283 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3469067491840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249212201172480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75902883 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239269765678080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270718021003264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592›ï‘d2ï- _wandbruntime‚3169  _runtime‚3169 Ñ{?û&ðõM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245984944421376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137219188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75901108 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 trainer/global_step‚3499 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137242325 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36043590 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274263940538880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242620788896217 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61318082 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3284222468569 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54264396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245984944421376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61318082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239336583467008 _runtime‚3169.658367749 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245984927250944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3296832980480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788443452610263e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137242325 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137219188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249282560540160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245984927250944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274263730020352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61318080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75924243 _step‚284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75901108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249282560540160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36043593 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3297616118784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61318080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470051599360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270793703193600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270793678420992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54264921 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3470237345280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270793703193600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270793678420992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54264921 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3470237345280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245984944421376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137219188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75901108 MGDevjvXÔ¡iceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 trainer/global_step‚3499 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137242325 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36043590 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274263940538880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242620788896217 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61318082 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3284222468569 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54264396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245984944421376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61318082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800308736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239336583467008 _runtime‚3169.658367749 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245984927250944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3296832980480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788443452610263e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137242325 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137219188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249282560540160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245984927250944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274263730020352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61318080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75924243 _step‚284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75901108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249282560540160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36043593 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3297616118784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61318080œ¹n‚2ò- _wandbruntime‚3169  _runtime‚3169—µ\¼WóÑ _runtime‚3169.659594645 _step‚285 0loss/FRt4/assignment_loss‚2.9119915432400174 epoch‚2 2loss/SRqqt1/assignment_loss‚2.2006681623255333 .loss/FRt2/detection_loss‚1.923224075360501 .loss/FRt3/detection_loss‚3.293739947179953 /loss/SRqqt3/detection_loss‚2.54366357984214 $ _timestamp‚1.788443452612416e+09 0loss/FRt2/assignment_loss‚2.9092091119035763 1loss/SRqqt2/assignment_loss‚2.147089696124844 1loss/FBt3/assignment_loss‚0.43422416605920133 /loss/FBt2/detection_loss‚0.5583486951712581 /loss/FRt3/assignment_loss‚4.832012703021367 0loss/FBt2/assignment_loss‚0.2681587931405665 0loss/SRqqt1/detection_loss‚2.198860938956098 /loss/FBt3/detection_loss‚2.8136559213910783 1loss/SRqqt3/assignment_loss‚2.293663083479322 1loss/FBt4/assignment_loss‚0.39319968646071046 0loss/FBt1/assignment_loss‚0.3704718626104295 trainer/global_step‚3499 /loss/FRt1/assignment_loss‚4.255614880153111 2loss/SRqqt4/assignment_loss‚1.9362539743432905 /loss/FBt1/detection_loss‚3.1379761919379234 %loss/total_loss‚53.73707433422405 /loss/FRt1/detection_loss‚2.9055830870355877 /loss/FRt4/detection_loss‚2.0385877076122494 1loss/SRqqt2/detection_loss‚2.5430497183747915 1loss/SRqqt4/detection_loss‚1.2209568554812138 /loss/FBt4/detection_loss‚3.6068699530192783ºÃ‰ì\ôÖ 0loss/FBt1/assignment_loss‚0.3704718626104295 trainer/global_step‚3499 /loss/FRt1/assignment_loss‚4.255614880153111 2loss/SRqqt4/assignment_loss‚1.9362539743432905 /loss/FBt1/detection_loss‚3.1379761919379234 %loss/total_loss‚53.73707433422405 /loss/FRt1/detection_loss‚2.9055830870355877 /loss/FRt4/detection_loss‚2.0385877076122494 1loss/SRqqt2/detection_loss‚2.5430497183747915 1loss/SRqqt4/detection_loss‚1.2209568554812138 /loss/FBt4/detection_loss‚3.6068699530192783 _runtime‚3169.659594645 _step‚285 0loss/FRt4/assignment_loss‚2.9119915432400174 epoch‚2 2loss/SRqqt1/assignment_loss‚2.2006681623255333 .loss/FRt2/detection_loss‚1.923224075360501 .loss/FRt3/detection_loss‚3.293739947179953 /loss/SRqqt3/detection_loss‚2.54366357984214 $ _timestamp‚1.788443452612416e+09 0loss/FRt2/assignment_loss‚2.9092091119035763 1loss/SRqqt2/assignment_loss‚2.147089696124844 1loss/FBt3/assignment_loss‚0.43422416605920133 /loss/FBt2/detection_loss‚0.5583486951712581 /loss/FRt3/assignment_loss‚4.832012703021367 0loss/FBt2/assignment_loss‚0.2681587931405665 0loss/SRqqt1/detection_loss‚2.198860938956098 /loss/FBt3/detection_loss‚2.8136559213910783 1loss/SRqqt3/assignment_loss‚2.293663083479322 1loss/FBt4/assignment_loss‚0.39319968646071046\ÏU(žõ‚(:“ ÆöåÔÍù¶Ç gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=)Vö‚(:Ë ÆöåÔ¨ÏÂÉ disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 567.71484375' disk.nvme0c0n1.out‚13274.9345703125 cpu‚25.23346728582708 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚13274.9345703125" disk.dm-0.out‚13274.9345703125 disk.drbd2265.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2727.84765625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125* disk./.usagePercent‚19.703932201995762$ disk./.usageGB‚293.6028289794922 disk.sda3.in‚0 disk.dm-1.out‚ 117.6484375 disk.drbd2080.in‚0* proc.memory.percent‚16.649460792541504 network.sent‚ 7.903244e+06 disk.sda3.out‚ 117.6484375 disk.rbd1.in‚0 disk.rbd1.out‚ 22.5517578125 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚ 1.134071e+06 disk.sda.in‚0 disk.sda.out‚ 117.6484375$ memory_percent‚96.53494358062744~úu“÷‚(:ˆ ÕöåÔÇÛ¯È gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¼8z\ø‚(:Ñ ÕöåÔÚÐÖÊ disk.nvme0c0n1.in‚0.078125 disk.drbd2266.in‚0( proc.memory.availableMB‚ 567.96484375 disk.rbd1.out‚ 22.6142578125 disk.sda2.out‚0# proc.memory.rssMB‚ 2727.84765625* proc.memory.percent‚16.649460792541504$ disk./.usageGB‚293.5808792114258 disk.drbd2080.out‚0 cpu‚25.23259390565551& disk.nvme0n1.out‚13320.11572265625* disk./.usagePercent‚19.702459134643743 disk.drbd2265.in‚0 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚13320.11572265625 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.dm-1.out‚ 118.21484375# disk.dm-0.out‚13320.11572265625 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 118.21484375 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0$ memory_percent‚96.53341770172119 network.sent‚ 8.011805e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 118.21484375 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1716.in‚0 network.recv‚ 1.139833e+06 disk.sda1.in‚0 disk.sda1.out‚0Ρ.žù‚(:“ äöåÔŸ¦¬Ç gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.481' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.696200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½RúZú‚(:Ï äöåÔÓÜãÊ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 118.6171875 disk.sda3.out‚ 118.6171875 network.recv‚ 1.144192e+06 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0# proc.memory.rssMB‚ 2727.84765625 disk.drbd2080.out‚0 disk.rbd1.out‚ 22.7080078125 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125* disk./.usagePercent‚19.703061520371808 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 118.6171875# disk.dm-0.out‚13355.15478515625 network.sent‚ 8.017673e+06 disk.drbd2265.in‚0 disk.dm-0.in‚0.078125* proc.memory.percent‚16.649460792541504 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚13355.15478515625 disk.sda.in‚0& disk.nvme0n1.out‚13355.15478515625$ memory_percent‚96.53799533843994 proc.cpu.threads‚17 cpu‚25.231746193969755 disk.sda2.out‚0( proc.memory.availableMB‚ 567.21484375 disk.dm-1.in‚0 disk.drbd1716.in‚0$ disk./.usageGB‚293.5898551940918 disk.drbd2266.in‚0 disk.drbd2080.in‚0»±‰2û- _wandbruntime‚3213  _runtime‚3213Ê]²Èü ' lr-AdamW/pg1‚0.00013712072880369672 ' lr-AdamW/pg2‚0.00013712072880369672 trainer/global_step‚3549 $ _timestamp‚1.788443496007022e+09 _runtime‚3213.054252863 _step‚286ùÙöÈÍýÇ _step‚286 ' lr-AdamW/pg1‚0.00013712072880369672 ' lr-AdamW/pg2‚0.00013712072880369672 trainer/global_step‚3549 $ _timestamp‚1.788443496007022e+09 _runtime‚3213.054252863ž'("Ù2þ- _wandbruntime‚3213  _runtime‚3213B–¯\ñ'ÿëO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274502105747456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252690168533504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62153040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223114752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3330291124333 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18466661 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚ 3213.05526313 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278020377984512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3518049122304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3343077305856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242610638085120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252690968828928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249347874352640 _step‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3342294180864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252690968828928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252690168533504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18466123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3343077305856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249347891523072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76949826 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198342144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62153042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3342294180864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54999321 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3518247464448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76972935xøÂE€ðO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252690168533504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62153040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223114752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3330291124333 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18466661 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚ 3213.05526313 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278020377984512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3518049122304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3343077305856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242610638085120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252690968828928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249347874352640 _step‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3342294180864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252690968828928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252690168533504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18466123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3343077305856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249347891523072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76949826 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198342144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62153042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3342294180864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54999321 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.Ê`‡†± peak‚1031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3518247464448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76972935 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274502105747456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111Ÿ'g¯2- _wandbruntime‚3213  _runtime‚3213Ÿ Ö´û&‚õM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55014775 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36543193 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3519038192128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18471582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252760527901184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249416505926144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252759727592448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62170080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3343221666304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18471057 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246007889754025 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139164377 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62170080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55014247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249416523096576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76971160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249416523096576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252760527901184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139164377 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62170080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55014247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249416523096576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76971160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249416523096576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252760527901184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36543193 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3519038192128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18471582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252760527901184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249416505926144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252759727592448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62170080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3343221666304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18471057 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246007889754025 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256168135894528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281853337422336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281853131084800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376502535741 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚256168135894528 PCDeviceStatsMonitor.on_traEpíôin_batch_start/allocation.small_pool.freed‚78019878 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252779453027840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249328047106621 _runtime‚3257.458076527 _step‚291 trainer/global_step‚3599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389465991680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55749304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18717034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63005042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18716506 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3567097385472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141024918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278286240036864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252779470198272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388682866688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245951544570880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78042987 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78042987 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389465991680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252779470198272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63005040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚256168936189952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245951527531520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55749835 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3375721334981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256168936189952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63005040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388682866688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37032801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141048029 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63005042 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424“ºŽ;÷'’ñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141048029 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63005042 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37032798 % _timestamp‚1.7884435404106224e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278286215264256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256168135894528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013120512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281853337422336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021187584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23111 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281853131084800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376502535741 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚256168135894528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78019878 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23111 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252779453027840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249328047106621 _runtime‚3257.458076527 _step‚291 trainer/global_step‚3599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389465991680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27967 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55749304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18717034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783124992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021187584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63005042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18716506 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3567097385472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141024918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278286240036864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252779470198272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030861 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388682866688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245951544570880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78042987 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78042987 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389465991680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252779470198272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63005040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27967 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚256168936189952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245951527531520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55749835 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3375721334981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256168936189952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63005040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3388682866688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37032801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013120512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783124992£r,!2“- _wandbruntime‚3258  _runtime‚3258žc¤=ü&”öM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278361922227200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278361897454592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189940224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281930004526592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246018345320448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249394990611833 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141086429 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141063292 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141086429 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256238495262208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78064347 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252848101771776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3567892359168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63022080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390393490432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 trainer/global_step‚3599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78041212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252848084601344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256237694953472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _step‚292 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252848101771776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141063292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37042790 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246018362359808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390393490432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389610352128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55764878 % _timestamp‚1.7884435412379022e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3376645291385 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78041212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55764346 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252848084601344@‹‚Ä•ûM _runtime‚3258.285735031 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78064347 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23135 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013120512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141063292 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800308736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030861 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141086429 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27967 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013120512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23137 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256238495262208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78064347 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252848101771776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.curÿŸ÷C=rent‚521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3567892359168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225933 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63022080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390393490432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 trainer/global_step‚3599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78041212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252848084601344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256237694953472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _step‚292 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252848101771776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141063292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37042790 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246018362359808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390393490432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389610352128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021187584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55764878 % _timestamp‚1.7884435412379022e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3376645291385 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78041212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55764346 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252848084601344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278361922227200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278361897454592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189940224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281930004526592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246018345320448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249394990611833 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141086429 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712832 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12259950592 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 12013120512 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚256911924872704 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚79022130 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3613673680384 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 605072384 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚260346438352384 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 603026004 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27967 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28216 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1019225933 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 3434513479680 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚249918058073088 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚190 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚286277970478592 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3613525481984 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚19100161 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 143128400 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚256911924872704 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18280 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚56821934 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12259950592 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚79022130 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1021187584 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚37721773 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3433925577728 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143128400 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143146682 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1031 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚521 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11635030861 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚260345833280000 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 587901952 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27967 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚260345833280000 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚256911907702272 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚37721776 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3433925577728 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1051 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12259950592 _runtime‚3334.805895784 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚64124552 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884436177579541e+09 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12259950592 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 12013120512 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚256911924872704 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚79022130 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3613673680384 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 605072384 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚260346438352384 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 603026004 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27967 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28216 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1019225933 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 3434513479680 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371895808 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚249918058073088 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚190 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚286277970478592 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3613525481984 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚19100161 \ODeviceStatsMonitor.on_validatioÎáw ín_batch_start/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 143128400 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚256911924872704 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18280 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚56821934 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12259950592 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚79022130 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1021187584 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚37721773 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3433925577728 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143128400 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143146682 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1031 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚521 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11635030861 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚260345833280000 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 587901952 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27967 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚260345833280000 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚256911907702272 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚37721776 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3433925577728 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1051 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12259950592 _runtime‚3334.805895784 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚64124552 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884436177579541e+09 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚56835171 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28216 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚64137660 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚189 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28216 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚286324460550656 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 605072384 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1010 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3434314518528 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18282 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚37730124 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 143167574 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚256957042640384 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚19105047 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12259950592 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27967 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3434314518528 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 _runtime‚ 3335.81490422 _step‚295 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚64137662 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1031 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚253382003072309 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚256957042640384 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 585986644 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18282 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18280 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 143167574 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚79029912 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 603026004 % _timestamp‚1.7884436187671382e+09 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚256957025469952 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3434902420480 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 143149292 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚192 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 172971008 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587901952 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3614062621184 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚56835171 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28216 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚64137660 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚189 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28216 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚286324460550656 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 605072384 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1010 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3434314518528 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18282 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚37730124 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 143167574 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚256957042640384 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚19105047 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12259950592 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27967 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3434314518528 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 _runtime‚ 3335.81490422 _step‚295 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚64137662 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1031 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚253382003072309 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚256957042640384 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 585986644§§¬ÑÃ’©‚(:‡ ë÷åÔ–ÕÇ gpu.0.gpu‚54.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¸d4Zª‚(:Ï ë÷åÔ‰ä÷É disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 cpu‚25.310358307179882$ disk./.usageGB‚293.7057456970215 disk.sda.out‚ 123.1796875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 123.1796875& disk.nvme0n1.out‚13714.41748046875 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.sda1.out‚0 network.recv‚ 1.184846e+06* disk./.usagePercent‚19.710839029262033( disk.nvme0c0n1.out‚13714.41748046875 disk.nvme0n1.in‚0.078125 disk.rbd1.out‚ 23.3271484375$ memory_percent‚96.53513431549072( proc.memory.availableMB‚ 567.68359375 network.sent‚ 8.276025e+06 disk.nvme0c0n1.in‚0.078125# disk.dm-0.out‚13714.41748046875 disk.rbd0.out‚0* proc.memory.percent‚16.643595695495605 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2726.88671875 disk.sda2.out‚0 disk.dm-1.out‚ 123.1796875R¤1p2«- _wandbruntime‚3349  _runtime‚3349½Éáh¬ʤ ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907019734382629 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.19004130363464355 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9611047506332397 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14393194019794464 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28649237751960754 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3272005319595337 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30307531356811523 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31414514780044556 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9378729462623596 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 % _timestamp‚1.7884436327817743e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964739680290222 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3099951446056366 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12836256623268127 +jet/accuracy_2_of_4‚0.28143635392189026 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11954618990421295 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4753153920173645 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666877269745 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 _step‚296 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015325039625168 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28128522634506226 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9885851144790649 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9808522462844849 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19758976995944977 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08020682632923126 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08409091085195541 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29073742032051086 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41414710879325867 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40447768568992615 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.28735634684562683 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.31090426445007324 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08811184763908386 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.25925925374031067 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *loss/val_total_loss‚52.256415066340445 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9870684742927551 ,&PurØÇo~ity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2775019109249115 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 jet/accuracy_5_of_6‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3678898513317108 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 jet/accuracy_4_of_6‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9102564454078674 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9872044324874878 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.059829238802194595 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1095038652420044 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14774054288864136 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2618405222892761 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0.5 particle/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992637038230896 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.334098219871521 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08306562900543213 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08949978649616241 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012817027047276497 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_5‚0.3560227155685425 ,particle/sensitivity‚0.24386900663375854 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003308193990960717 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33120477199554443 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.35595497488975525 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10460726171731949 particle/accuracy_2_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4077354669570923 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 (particle/f_score‚0.30865469574928284 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3346960246562958 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2906433343887329 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08737646043300629 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2905084192752838 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3632829785346985 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21212123334407806 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9964911937713623 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9921894669532776 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07399728149175644 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1464577317237854 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3333333134651184 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009788359515368938 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8228311538696289 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12484344840049744 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9909836053848267 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02339283376932144 *jet/accuracy_2_of_6‚0.2884615361690521 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9888269901275635 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3247474133968353 particle/accuracy_3_of_4‚1 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27987420558929443 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5299671292304993 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.39870694279670715 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 *jet/accuracy_1_of_5‚0.8715629577636719 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3819488286972046 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.23456786572933197 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19485677778720856 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16301247477531433 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06321855634450912 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3646898865699768 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03030303120613098 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015693718567490578 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16749227046966553 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.32828864455223083 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2855118215084076 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08592994511127472 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1270086169242859 particle/accuracy_1_of_3‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2604166567325592 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08731881529092789 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5305619835853577 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821645021438599 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.22962966561317444 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.28060999512672424 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3711150586605072 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15316590666770935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11346686631441116 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.30519023537635803 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_3_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4129829406738281 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 (particle/accuracy‚0.8295149207115173 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 +jet/accuracy_1_of_1‚0.40914732217788696 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.0771055519580841 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.44518882036209106 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5090277791023254 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3706463575363159 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2296649068593979 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008611230179667473 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1367245614528656 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3250082731246948 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031609196215867996 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16298705339431763 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9710129499435425 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12448558956384659 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.003513071918860078 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.40050339698791504 particle/accuracy_1_of_2‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34166666865348816 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15184824168682098 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.09189189225435257 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884928464889526 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07413309067487717 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10144177824258804 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9791396260261536 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09374188631772995 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3013215959072113 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34335049986839294 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979872107505798 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27837833762168884 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2743055522441864 particle/accuracy_3_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14482757449150085 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1597222238779068 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35739848017692566 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946340322494507 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08671274036169052 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.982508659362793 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.45280832052230835 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3077813982963562 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880064725875854 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07972937822341919 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0038548754528164864 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.009523809887468815 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09099673479795456 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3984169363975525 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44226938486099243 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3940894305706024 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.6511194705963135 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.007768082898110151 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4294692277908325 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4394840896129608 particle/accuracy_4_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 _runtime‚3349.831298441 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.34418562054634094 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07439816743135452 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02994634211063385 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6243538856506348 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.34332898259162903 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.524084210395813 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1299973875284195 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.12870369851589203 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07399728149175644 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.373653382062912 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13944198191165924 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.44255951046943665 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4167117178440094 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044238075613975525 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3782423138618469 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36169102787971497 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.43581318855285645 *jet/accuracy_1_of_3‚0.7593708634376526 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12267912179231644 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2948412597179413 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3819488286972046 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8204298615455627 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.46606889367103577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918832182884216 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5046610236167908 particle/accuracy_2_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3175804018974304 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003477357095107436 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.038387805223464966 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,jet/accuracy_3_of_3‚0.012973363511264324 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚3626 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09712642431259155 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.44012340903282166 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.032439347356557846 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2883492708206177 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07473991811275482 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13333334028720856 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4537036716938019 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6391329169273376 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20919539034366608 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 epoch‚2 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.29051727056503296 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28251487016677856 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19911786913871765 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03860510513186455 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003514545038342476 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.368059366941452 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.33274438977241516 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9836709499359131 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.38424909114837646 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3649332821369171 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023579109460115433 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3008928596973419 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12372922152280807 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32578280568122864 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005376344081014395 particle/accuracy_5_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5529570579528809 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_6_of_6‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2868216931819916 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0ˆ.oc­Ϥ A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.0771055519580841 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.44518882036209106 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5090277791023254 @&Purity/1FRt3SRqqt2FBt/event_prô^ó·ùoportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3706463575363159 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2296649068593979 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008611230179667473 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1367245614528656 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3250082731246948 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031609196215867996 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16298705339431763 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9710129499435425 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12448558956384659 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.003513071918860078 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.40050339698791504 particle/accuracy_1_of_2‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34166666865348816 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15184824168682098 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.09189189225435257 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884928464889526 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07413309067487717 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10144177824258804 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9791396260261536 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09374188631772995 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3013215959072113 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34335049986839294 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979872107505798 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27837833762168884 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2743055522441864 particle/accuracy_3_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14482757449150085 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1597222238779068 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35739848017692566 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946340322494507 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08671274036169052 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.982508659362793 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.45280832052230835 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3077813982963562 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880064725875854 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07972937822341919 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0038548754528164864 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.009523809887468815 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09099673479795456 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3984169363975525 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.44226938486099243 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3940894305706024 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.6511194705963135 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.007768082898110151 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4294692277908325 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4394840896129608 particle/accuracy_4_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 _runtime‚3349.831298441 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.34418562054634094 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07439816743135452 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02994634211063385 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6243538856506348 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.34332898259162903 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.524084210395813 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1299973875284195 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.12870369851589203 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07399728149175644 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.373653382062912 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13944198191165924 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.44255951046943665 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4167117178440094 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.044238075613975525 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3782423138618469 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36169102787971497 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.43581318855285645 *jet/accuracy_1_of_3‚0.7593708634376526 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12267912179231644 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2948412597179413 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3819488286972046 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8204298615455627 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.46606889367103577 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918832182884216 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5046610236167908 particle/accuracy_2_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3175804018974304 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003477357095107436 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.038387805223464966 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,jet/accuracy_3_of_3‚0.012973363511264324 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚3626 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09712642431259155 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.44012340903282166 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.032439347356557846 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2883492708206177 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07473991811275482 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13333334028720856 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4537036716938019 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6391329169273376 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20919539034366608 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 epoch‚2 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.29051727056503296 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.28251487016677856 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19911786913871765 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03860510513186455 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003514545038342476 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.368059366941452 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.33274438977241516 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9836709499359131 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.38424909114837646 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3649332821369171 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023579109460115433 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3008928596973419 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12372922152280807 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32578280568122864 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005376344081014395 particle/accuracy_5_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5529570579528809 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_6_of_6‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2868216931819916 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907019734382629 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.19004130363464355 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9611047506332397 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14393194019794464 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28649237751960754 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3272005319595337 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30307531356811523 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31414514780044556 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9378729462623596 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 % _timestamp‚1.7884436327817743e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964739680290222 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3099951446056366 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12836256623268127 +jet/accuracy_2_of_4‚0.28143635392189026 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11954618990421295 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4753153920173645 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666877269745 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 _step‚296 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015325039625168 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28128522634506226 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9885851144790649 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9808522462844849 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19758976995944977 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08020682632923126 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08409091085195541 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29073742032051086 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41414710879325867 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40447768568992615 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.28735634684562683 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.31090426445007324 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08811184763908386 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.25925925374031067 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *loss/val_total_loss‚52.256415066340445 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9870684742927551 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2775019109249115 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 jet/accuracy_5_of_6‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3678898513317108 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 jet/accuracy_4_of_6‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9102564454078674 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9872044324874878 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.059829238802194595 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1095038652420044 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14774054288864136 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2618405222892761 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0.5 particle/accuracy_1_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992637038230896 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.334098219871521 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08306562900543213 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08949978649616241 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012817027047276497 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_2_of_5‚0.3560227155685425 ,particle/sensitivity‚0.24386900663375854 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003308193990960717 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33120477199554443 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.35595497488975525 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10460726171731949 particle/accuracy_2_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4077354669570923 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 (particle/f_score‚0.30865469574928284 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3346960246562958 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2906433343887329 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0026455027982592583 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08737646043300629 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2905084192752838 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3632829785346985 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21212123334407806 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9964911937713623 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9921894669532776 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07399728149175644 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1464577317237854 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3333333134651184 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009788359515368938 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8228311538696289 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12484344840049744 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9909836053848267 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02339283376932144 *jet/accuracy_2_of_6‚0.2884615361690521 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9888269901275635 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3247474133968353 particle/accuracy_3_of_4‚1 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27987420558929443 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5299671292304993 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.39870694279670715 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 *jet/accuracy_1_of_5‚0.8715629577636719 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3819488286972046 97éûØú"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.23456786572933197 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19485677778720856 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16301247477531433 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06321855634450912 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3646898865699768 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03030303120613098 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015693718567490578 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16749227046966553 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.32828864455223083 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2855118215084076 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08592994511127472 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1270086169242859 particle/accuracy_1_of_3‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2604166567325592 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08731881529092789 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5305619835853577 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821645021438599 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.22962966561317444 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.28060999512672424 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3711150586605072 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15316590666770935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11346686631441116 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.30519023537635803 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_3_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4129829406738281 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 (particle/accuracy‚0.8295149207115173 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 +jet/accuracy_1_of_1‚0.40914732217788696 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0¨ŸªA“®‚(:ˆ ú÷åÔ‚œíÈ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùcé_V¯‚(:Ë ú÷åÔ€‚Ë disk.dm-0.in‚0.078125 cpu‚25.312655117267788 disk.rbd0.in‚0 disk.sda.out‚ 123.53125 disk.sda2.out‚0$ memory_percent‚93.06089878082275 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚ 123.53125 disk.drbd2266.out‚0* proc.memory.percent‚16.643357276916504 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 network.sent‚ 8.385243e+06% disk./.usageGB‚294.21575927734375 disk.drbd2265.out‚0 network.recv‚ 1.191573e+06( disk.nvme0c0n1.out‚14323.45849609375 disk.sda3.in‚0# disk.dm-0.out‚14323.45849609375 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 123.53125 disk.rbd1.out‚ 23.4677734375 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1136.90234375# proc.memory.rssMB‚ 2726.84765625* disk./.usagePercent‚19.745066468567355 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125& disk.nvme0n1.out‚14323.45849609375UôR6ž°‚(:“ ‰øåÔçŽÔÇ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.367399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`JW±‚(:Ì ‰øåÔîïÉ disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125# proc.memory.rssMB‚ 2725.86328125 disk.drbd2266.out‚0 network.recv‚ 1.19734e+06 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 124.41796875 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 cpu‚25.311517574874085% disk./.usageGB‚294.24170303344727% disk.nvme0n1.out‚14927.7646484375" disk.dm-0.out‚14927.7646484375 disk.sda3.in‚0* disk./.usagePercent‚19.746807575807722 disk.sda.in‚0 disk.dm-1.out‚ 124.41796875( proc.memory.availableMB‚ 571.37890625 network.sent‚ 8.508543e+06 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda3.out‚ 124.41796875 disk.dm-1.in‚0$ memory_percent‚96.51257991790771* proc.memory.percent‚16.637349128723145 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 23.7734375' disk.nvme0c0n1.out‚14927.7646484375š‚L2²- _wandbruntime‚3375  _runtime‚3375DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3446354489346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262513042879488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259053468237312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284974720823296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3447135690110 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚298 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64682970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255433156047874 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3460374938112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255433954287998 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259053451066880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79582131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288615389956096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262513843175424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38058511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259053451066880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144288213 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262513843175424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79605241 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251986801558528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3459591812608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3459591812608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284974696050688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3640669132800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57336790 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64682972Í¥ö'·ðO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79582131 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3459591812608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3459591812608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284974696050688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3640669132800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57336790 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64682972 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3460374938112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64682970 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57336275 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79605241 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251986818597888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚3649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3446354489346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262513042879488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259053468237312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284974720823296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3447135690110 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚298 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64682970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255433156047874 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3460374938112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255433954287998 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259053451066880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79582131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288615389956096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262513843175424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38058511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259053451066880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144288213 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262513843175424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79605241 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251986801558528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3376.327097645 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252053636386816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144303475 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259122082640384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252053619347456 _step‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144326613 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255500897793206 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 $ _timestamp‚1.788443659279251e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262582601938432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288692056488448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3460519298048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚3649 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64700012 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262583402247680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255501696033538 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144326613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285050403013632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206323712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288691850164736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3447278445750 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259122099810816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79626601 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79603465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64700012 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57351065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3448059646722 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3461302436864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79626601 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259122099810816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181551104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3460519298048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64700010 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79603465 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328+bcÚ'ºúM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144326613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285050403013632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206323712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288691850164736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3447278445750 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259122099810816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79626601 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79603465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64700012 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57351065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3448059646722 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3461302436864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79626601 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259122099810816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181551104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3460519298048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64700010 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79603465 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262583402247680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144303475 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38068506 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285050378241024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19282562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3376.327097645 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252053636386816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144303475 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259122082640384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252053619347456 _step‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144326613 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255500897793206 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 $ _timestamp‚1.788443659279251e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262582601938432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288692056488448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3460519298048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚3649 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64700012 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262583402247680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255501696033538«¿ÄT2»- _wandbruntime‚3376  _runtime‚3376œû{ R¼Ì /loss/FRt4/assignment_loss‚3.710816083952438 1loss/SRqqt4/assignment_loss‚2.026185882139079 0loss/FBt3/assignment_loss‚0.5275570938629244 .loss/FRt1/detection_loss‚2.607875320173445 -loss/FBt3/detection_loss‚2.01299717596599 _step‚300 2loss/SRqqt1/assignment_loss‚1.8077300791831121 1loss/FBt2/assignment_loss‚0.25332582569863554 0loss/SRqqt2/detection_loss‚1.861781114765576 1loss/SRqqt4/detection_loss‚1.4309776298146932 0loss/FRt3/assignment_loss‚4.1669101030738265 .loss/FRt2/detection_loss‚2.330457130793569 /loss/FBt4/detection_loss‚3.1533995792269707 1loss/SRqqt2/assignment_loss‚2.117278385481664 1loss/SRqqt1/detection_loss‚2.2322445842234986 .loss/FBt1/detection_loss‚4.451121985912323 2loss/SRqqt3/assignment_loss‚1.8687868186405727 0loss/FBt1/assignment_loss‚1.1232117116451263 /loss/FRt4/detection_loss‚2.5619652364836183 % _timestamp‚1.7884436592811337e+09 _runtime‚3376.328267033 %loss/total_loss‚53.80032104964191 epoch‚3 trainer/global_step‚3649 0loss/FBt4/assignment_loss‚0.4561263485811651 .loss/FRt3/detection_loss‚3.169005405019831 /loss/SRqqt3/detection_loss‚2.69731183052063 /loss/FBt2/detection_loss‚0.9456454709598661 /loss/FRt1/assignment_loss‚3.609175200973238 /loss/FRt2/assignment_loss‚2.678435052550116)"ÍW½Ñ 0loss/FRt3/assignment_loss‚4.1669101030738265 .loss/FRt2/detection_loss‚2.330457130793569 /loss/FBt4/detection_loss‚3.1533995792269707 1loss/SRqqt2/assignment_loss‚2.117278385481664 1loss/SRqqt1/detection_loss‚2.2322445842234986 .loss/FBt1/detection_loss‚4.451121985912323 2loss/SRqqt3/assignment_loss‚1.8687868186405727 0loss/FBt1/assignment_loss‚1.1232117116451263 /loss/FRt4/detection_loss‚2.5619652364836183 % _timestamp‚1.7884436592811337e+09 _runtime‚3376.328267033 %loss/total_loss‚53.80032104964191 epoch‚3 trainer/global_step‚3649 0loss/FBt4/assignment_loss‚0.4561263485811651 .loss/FRt3/detection_loss‚3.169005405019831 /loss/SRqqt3/detection_loss‚2.69731183052063 /loss/FBt2/detection_loss‚0.9456454709598661 /loss/FRt1/assignment_loss‚3.609175200973238 /loss/FRt2/assignment_loss‚2.678435052550116 /loss/FRt4/assignment_loss‚3.710816083952438 1loss/SRqqt4/assignment_loss‚2.026185882139079 0loss/FBt3/assignment_loss‚0.5275570938629244 .loss/FRt1/detection_loss‚2.607875320173445 -loss/FBt3/detection_loss‚2.01299717596599 _step‚300 2loss/SRqqt1/assignment_loss‚1.8077300791831121 1loss/FBt2/assignment_loss‚0.25332582569863554 0loss/SRqqt2/detection_loss‚1.861781114765576 1loss/SRqqt4/detection_loss‚1.4309776298146932¬c¸¾"“¾‚(:ˆ ˜øåÔ禩Ç gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûï¨$L¿‚(:Á ˜øåÔîƼÉ network.sent‚ 8.514599e+06$ disk./.usageGB‚294.2504768371582 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 124.859375 disk.sda3.in‚0 network.recv‚ 1.201934e+06 disk.rbd1.out‚ 23.7734375 disk.sda1.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.86328125 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 124.859375* proc.memory.percent‚16.637349128723145 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0$ memory_percent‚96.51432037353516 cpu‚25.3103022876212% disk.nvme0n1.out‚14962.1904296875" disk.dm-0.out‚14962.1904296875 disk.rbd1.in‚0* disk./.usagePercent‚19.747396393136473 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 124.859375' disk.nvme0c0n1.out‚14962.1904296875% proc.memory.availableMB‚ 571.09375& žÀ‚(:“ §øåÔˆÊÇ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø¿ëVÁ‚(:Ë §øåÔ¦úÆÉ disk.drbd1716.out‚0 disk.drbd2266.out‚0# memory_percent‚95.7742691040039 disk.dm-1.out‚ 125.43359375 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.637349128723145 proc.cpu.threads‚17 disk.dm-1.in‚0% disk.nvme0n1.out‚15015.6455078125 disk.sda1.out‚0$ disk./.usageGB‚294.2694320678711 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚15015.6455078125 disk.sda3.in‚0 disk.rbd1.out‚ 23.9853515625# proc.memory.rssMB‚ 2725.86328125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.recv‚ 1.208054e+06* disk./.usagePercent‚19.748668494574115 disk.sda.out‚ 125.43359375 disk.sda3.out‚ 125.43359375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2080.out‚0% proc.memory.availableMB‚ 692.34375 cpu‚25.309235871019816 network.sent‚ 8.623185e+06 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚15015.6455078125 disk.drbd2266.in‚0JxCžÂ‚(:“ ¶øåÔ¬£²Ç gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OÀÊÍ^Â(:Ó ¶øåÔò»ÈÉ& disk.nvme0n1.out‚15055.52294921875 disk.dm-0.in‚0.078125# disk.dm-0.out‚15055.52294921875 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2725.86328125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15055.52294921875 disk.sda2.in‚0 disk.drbd2265.out‚0 cpu‚25.308034739663892 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 125.83203125 disk.sda.in‚0 disk.sda3.out‚ 125.83203125 disk.drbd2266.out‚0* disk./.usagePercent‚19.749075546553556 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 network.sent‚ 8.629512e+06% disk./.usageGB‚294.27549743652344 disk.sda3.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 691.83203125* proc.memory.percent‚16.637349128723145 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0$ memory_percent‚95.77739238739014 network.recv‚ 1.212768e+06 disk.rbd1.out‚ 24.3916015625 disk.dm-1.out‚ 125.83203125 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0îOo2Ä- _wandbruntime‚3420  _runtime‚3420Ô"NÔÇÅÁ ' lr-AdamW/pg2‚0.00013497901689455655 trainer/global_step‚3699 # _timestamp‚1.78844370301396e+09 _runtime‚3420.061216915 _step‚301 ' lr-AdamW/pg1‚0.00013497901689455655Òê–_ÌÆÆ ' lr-AdamW/pg1‚0.00013497901689455655 ' lr-AdamW/pg2‚0.00013497901689455655 trainer/global_step‚3699 # _timestamp‚1.78844370301396e+09 _runtime‚3420.061216915 _step‚301­šeÖ%2Ç- _wandbruntime‚3420  _runtime‚3420’jpc­#ÈìO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3689194341888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80652183 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80675293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210531328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚262485046912512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚262485046912512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19524389 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255327691004928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3493347101518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3492565900754 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65534972 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146210265 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258821055145806 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255327708044288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 _runtime‚3420.062725162 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262485029742080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3505980498432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146187153 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3506763623936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262485029742080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265991010240512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 % _timestamp‚1.7884437030150266e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65534972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38558114 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3689008583168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80675293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19523870 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265991010240512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 H>DeviceStat{¥–¥EsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3506763623936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265991010240512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3506763623936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262485029742080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3505980498432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146187153 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3506763623936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262485029742080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚302 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265991010240512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 % _timestamp‚1.7884437030150266e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65534972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38558114®‘š[2Ê- _wandbruntime‚3420  _runtime‚3420DÑ 6ü&ËöM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80696653 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3506907983872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3690164843520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258888796891346 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262553678486016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262553661315584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58097358 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146248665 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146248665 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65552010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80696653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19529252 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292524454273024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288834512530432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80673517 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19528714 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3507691122688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3420.902951601 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262553678486016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65552012 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292524677373952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266061369608704 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3494271058130 % _timestamp‚1.7884437038550363e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146225527 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58096817 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255394525833216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255394508793856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0q^J×'ÌûM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258888796891346 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262553678486016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262553661315584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58097358 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146248665 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146248665 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65552010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80696653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19529252 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292524454273024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288834512530432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80673517 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19528714 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3507691122688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3420.902951601 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262553678486016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65552012 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292524677373952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266061369608704 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3494271058130 % _timestamp‚1.7884437038550363e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146225527 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58096817 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255394525833216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255394508793856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80696653 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3506907983872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3690164843520¯ãPðA2Í- _wandbruntime‚3420  _runtime‚3420hœO2ÎÓ &loss/total_loss‚51.540233310248965 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚1.7319561700026194 /loss/FBt3/detection_loss‚2.7959170281887054 % _timestamp‚1.7884437038569148e+09 0loss/FRt2/assignment_loss‚3.2903156789337715 2loss/SRqqt3/assignment_loss‚2.0808051816726985 0loss/FBt3/assignment_loss‚0.3213947567347759 1loss/SRqqt1/detection_loss‚1.9360046486059825 1loss/SRqqt4/detection_loss‚1.4795828844704058 /loss/FRt1/assignment_loss‚4.687324474497539 2loss/SRqqt1/assignment_loss‚1.8165918929236275 0loss/FBt2/assignment_loss‚0.2681965651209209 .loss/FRt4/detection_loss‚2.475323703060759 epoch‚3 _step‚304 1loss/SRqqt2/detection_loss‚1.7534397731224696 0loss/SRqqt3/detection_loss‚2.464586915154206 2loss/SRqqt4/assignment_loss‚1.7403697629013788 1loss/FBt1/assignment_loss‚0.07780070323471477 /loss/FBt2/detection_loss‚0.6479012111560407 /loss/FBt4/detection_loss‚3.5536823206477695 _runtime‚3420.904061677 /loss/FRt3/assignment_loss‚4.230000298363822 0loss/FBt4/assignment_loss‚0.6916804489123428 /loss/FRt1/detection_loss‚2.5565925952864856 .loss/FRt2/detection_loss‚2.207729491518765 .loss/FBt1/detection_loss‚3.180842846632004 0loss/FRt4/assignment_loss‚2.9538516034471227 -loss=2Ê'/FRt3/detection_loss‚2.59834235566003E¹Xt^ÏØ 2loss/SRqqt4/assignment_loss‚1.7403697629013788 1loss/FBt1/assignment_loss‚0.07780070323471477 /loss/FBt2/detection_loss‚0.6479012111560407 /loss/FBt4/detection_loss‚3.5536823206477695 _runtime‚3420.904061677 /loss/FRt3/assignment_loss‚4.230000298363822 0loss/FBt4/assignment_loss‚0.6916804489123428 /loss/FRt1/detection_loss‚2.5565925952864856 .loss/FRt2/detection_loss‚2.207729491518765 .loss/FBt1/detection_loss‚3.180842846632004 0loss/FRt4/assignment_loss‚2.9538516034471227 -loss/FRt3/detection_loss‚2.59834235566003 &loss/total_loss‚51.540233310248965 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚1.7319561700026194 /loss/FBt3/detection_loss‚2.7959170281887054 % _timestamp‚1.7884437038569148e+09 0loss/FRt2/assignment_loss‚3.2903156789337715 2loss/SRqqt3/assignment_loss‚2.0808051816726985 0loss/FBt3/assignment_loss‚0.3213947567347759 1loss/SRqqt1/detection_loss‚1.9360046486059825 1loss/SRqqt4/detection_loss‚1.4795828844704058 /loss/FRt1/assignment_loss‚4.687324474497539 2loss/SRqqt1/assignment_loss‚1.8165918929236275 0loss/FBt2/assignment_loss‚0.2681965651209209 .loss/FRt4/detection_loss‚2.475323703060759 epoch‚3 _step‚304 1loss/SRqqt2/detection_loss‚1.7534397731224696 0loss/SRqqt3/detection_loss‚2.464586915154206°J HƓЂ(:ˆ ÅøåÔ¦Â¥È gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oiOZÑ‚(:Ï ÅøåÔËÆÎÊ disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 126.84765625 proc.cpu.threads‚17 network.sent‚ 8.635208e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda.out‚ 126.84765625 disk.sda2.out‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.637349128723145 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2725.86328125 disk.drbd2080.in‚0 disk.rbd1.out‚ 24.7197265625 disk.sda1.in‚0& proc.memory.availableMB‚ 691.765625 cpu‚25.3069731062913* disk./.usagePercent‚19.750133625692573 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125& disk.nvme0n1.out‚15099.40087890625 disk.sda2.in‚0 disk.drbd1716.out‚0 network.recv‚ 1.217027e+06% disk./.usageGB‚294.29126358032227 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15099.40087890625 disk.dm-0.in‚0.078125# disk.dm-0.out‚15099.40087890625$ memory_percent‚95.77779769897461 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 126.84765625/³žÒ‚(:“ ÔøåÔ“ùÂÇ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚136.892' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.378399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ÈŠ“]Ó‚(:Ò ÔøåÔà‰æÊ disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 691.57421875 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚15125.83056640625 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0% disk./.usageGB‚294.31482696533203# memory_percent‚95.7789659500122# proc.memory.rssMB‚ 2725.86328125* proc.memory.percent‚16.637349128723145 cpu‚25.305861271898177 network.sent‚ 8.744147e+06 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda2.in‚0* disk./.usagePercent‚19.751714984231576 disk.drbd2266.in‚0 disk.sda3.out‚ 127.24609375 proc.cpu.threads‚17 disk.sda.out‚ 127.24609375 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 1.223254e+06 disk.dm-1.out‚ 127.24609375# disk.dm-0.out‚15125.83056640625 disk.drbd2266.out‚0 disk.rbd1.out‚ 24.7197265625 disk.sda1.out‚0 disk.nvme0n1.in‚0.078125( disk.nvme0c0n1.out‚15125.83056640625fÐ{b“Ô‚(:ˆ ãøåÔ¹‘¨Ç gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÖSÕ‚(:È ãøåÔšñÉ disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 cpu‚25.304703956843074 proc.cpu.threads‚17% disk./.usageGB‚294.32617568969727 disk.drbd2080.out‚0 disk.sda3.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.77934741973877* proc.memory.percent‚16.637349128723145 disk.rbd0.out‚0% disk.nvme0c0n1.out‚15164.05859375) disk./.usagePercent‚19.75247660664594 disk.nvme0c0n1.in‚0.078125# disk.nvme0n1.out‚15164.05859375 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2725.86328125 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 691.51171875 disk.sda2.out‚0 disk.dm-1.out‚ 127.94140625 disk.dm-0.out‚15164.05859375 disk.drbd2266.out‚0 disk.rbd1.out‚ 24.814453125 disk.sda3.out‚ 127.94140625 disk.dm-0.in‚0.078125 network.sent‚ 8.750012e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 network.recv‚ 1.227658e+06 disk.sda1.in‚0 disk.sda.out‚ 127.94140625ÛsD2Ö- _wandbruntime‚3465  _runtime‚3465)Ï7îÉ×à ' lr-AdamW/pg1‚0.00013424075487573084 ' lr-AdamW/pg2‚0.00013424075487573084 trainer/global_step‚3749 % _timestamp‚1.7884437479863791e+09 _runtime‚3465.033624611 _step‚305ÚhnÎØÈ trainer/global_step‚3749 % _timestamp‚1.7884437479863791e+09 _runtime‚3465.033624611 _step‚305 ' lr-AdamW/pg1‚0.00013424075487573084 ' lr-AdamW/pg2‚0.00013424075487573084±>ßɤ2Ù- _wandbruntime‚3465  _runtime‚3465@íÓò'ÚìO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81745345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265916608417280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262208156003614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66386970 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552369184256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148132317 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19771506 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292542939856896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66386972 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58828684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269468977601536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269469777897472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81722235 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258668597490688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258668580451328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296280892632576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262207357763490 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269469777897472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265916625587712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269468977601536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265916625587712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39057711 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296280680004096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3737764919808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3553152309760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553152309760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81745345 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66386972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 % _timestamp‚1.7884437479871109e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552369184256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚3749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265916608417280ü¿RE÷'ÛñO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269468977601536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269469777897472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81722235 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258668597490688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258668580451328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296280892632576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262207357763490 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269469777897472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265916625587712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269468977601536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265916625587712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39057711 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296280680004096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3737764919808 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3553152309760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553152309760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81745345 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66386972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 % _timestamp‚1.7884437479871109e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3552369184256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚3749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265916608417280 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81745345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265916608417280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262208156003614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66386970 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552369184256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148132317 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19771506 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292542939856896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66386972 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58828684²¢Fb{2Ü- _wandbruntime‚3465  _runtime‚3465ñGÆ'ÝöM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58844159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269538536660480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58843619 _runtime‚3465.868006599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81766705 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265985257161216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269539336969728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 _step‚307 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148170717 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66404012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296357559444480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262275897749154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554079808512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148170717 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553296669696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024Ûïñ'ÞûM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148170717 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66404012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296357559444480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262275897749154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554079808512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148170717 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553296669696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58844159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269538536660480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58843619 _runtime‚3465.868006599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81766705 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265985257161216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269539336969728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 _step‚307³Ö1û12ß- _wandbruntime‚3465  _runtime‚3465è­Gð]à× 2loss/SRqqt1/assignment_loss‚2.1238998203557364 1loss/SRqqt4/assignment_loss‚2.068150875469049 trainer/global_step‚3749 _step‚308 1loss/SRqqt1/detection_loss‚1.8980070188337443 1loss/SRqqt4/detection_loss‚1.4763587475532578 .loss/FBt3/detection_loss‚2.908544673369481 /loss/FBt4/detection_loss‚3.0507378816604613 _runtime‚3465.869096435 .loss/FRt1/detection_loss‚2.590929002687335 2loss/SRqqt2/assignment_loss‚1.8949334463386824 1loss/FBt1/assignment_loss‚0.11548846028745174 /loss/FBt1/detection_loss‚3.6841498017311096 0loss/FRt4/assignment_loss‚3.6441824216591683 /loss/FRt2/detection_loss‚1.9385508798683684 /loss/FBt2/detection_loss‚0.8696716338950534 /loss/FRt1/assignment_loss‚3.977764345705509 2loss/SRqqt3/assignment_loss‚2.3493131339550017 1loss/FBt2/assignment_loss‚0.16868506943405076 epoch‚3 1loss/FBt3/assignment_loss‚0.44588952198395976 1loss/FBt4/assignment_loss‚0.41761791733601966 0loss/FRt2/assignment_loss‚2.8053058410684266 /loss/FRt3/assignment_loss‚4.260777950286865 /loss/FRt3/detection_loss‚2.8709062797682625 1loss/SRqqt2/detection_loss‚1.7654996265967686 1loss/SRqqt3/detection_loss‚2.7110292094094413 .loss/FRt4/detection_loss‚2.384872840423333 %loss/total_loss‚52.42126639967653 % _timestamp‚1.7884437488219645e+09J[báÜ 1loss/FBt2/assignment_loss‚0.16868506943405076 epoch‚3 1loss/FBt3/assignment_loss‚0.44588952198395976 1loss/FBt4/assignment_loss‚0.41761791733601966 0loss/FRt2/assignment_loss‚2.8053058410684266 /loss/FRt3/assignment_loss‚4.260777950286865 /loss/FRt3/detection_loss‚2.8709062797682625 1loss/SRqqt2/detection_loss‚1.7654996265967686 1loss/SRqqt3/detection_loss‚2.7110292094094413 .loss/FRt4/detection_loss‚2.384872840423333 %loss/total_loss‚52.42126639967653 % _timestamp‚1.7884437488219645e+09 2loss/SRqqt1/assignment_loss‚2.1238998203557364 1loss/SRqqt4/assignment_loss‚2.068150875469049 trainer/global_step‚3749 _step‚308 1loss/SRqqt1/detection_loss‚1.8980070188337443 1loss/SRqqt4/detection_loss‚1.4763587475532578 .loss/FBt3/detection_loss‚2.908544673369481 /loss/FBt4/detection_loss‚3.0507378816604613 _runtime‚3465.869096435 .loss/FRt1/detection_loss‚2.590929002687335 2loss/SRqqt2/assignment_loss‚1.8949334463386824 1loss/FBt1/assignment_loss‚0.11548846028745174 /loss/FBt1/detection_loss‚3.6841498017311096 0loss/FRt4/assignment_loss‚3.6441824216591683 /loss/FRt2/detection_loss‚1.9385508798683684 /loss/FBt2/detection_loss‚0.8696716338950534 /loss/FRt1/assignment_loss‚3.977764345705509 2loss/SRqqt3/assignment_loss‚2.3493131339550017´ °\¥“â‚(:ˆ òøåÔÛŒºÇ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.164' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0328 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*{cÒLã‚(:Á òøåÔó‚½Ê disk.drbd2080.in‚0 disk.dm-0.out‚15205.53515625 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚95.77999114990234# proc.memory.rssMB‚ 2725.86328125 disk.drbd2265.out‚0 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚15205.53515625 disk.dm-1.out‚ 128.41015625 cpu‚25.30360540672504 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125) disk./.usagePercent‚19.75444172047877 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 24.955078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125# disk.nvme0n1.out‚15205.53515625 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.sent‚ 8.756302e+06 network.recv‚ 1.2323e+06* proc.memory.percent‚16.637349128723145 disk.sda1.out‚0 disk.sda.out‚ 128.41015625 disk.sda2.in‚0 disk.drbd1716.in‚0% proc.memory.availableMB‚ 691.40625$ disk./.usageGB‚294.3554573059082 disk.sda3.out‚ 128.41015625 disk.drbd2266.in‚0«ó«žä‚(:“ ùåÔîûÐÇ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.542' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.108400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ßÏ÷Lå‚(:Á ùåÔü£çÉ$ disk.nvme0n1.out‚15335.966796875 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0! disk.dm-0.out‚15335.966796875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.86328125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 network.sent‚ 8.864479e+06 network.recv‚ 1.238166e+06 disk.rbd1.out‚ 25.017578125 disk.sda.out‚ 129.234375 disk.dm-1.out‚ 129.234375 disk.nvme0n1.in‚0.078125 disk.drbd2266.in‚0' proc.memory.availableMB‚ 691.3828125* proc.memory.percent‚16.637349128723145 proc.cpu.threads‚17& disk.nvme0c0n1.out‚15335.966796875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 cpu‚25.30253489779511) disk./.usagePercent‚19.75503591396574$ disk./.usageGB‚294.3643112182617 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 129.234375 disk.drbd2265.out‚0# memory_percent‚95.7801342010498 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0jÌø2æ- _wandbruntime‚3509  _runtime‚3509«L¨ÓÉçà _runtime‚3509.951868186 _step‚309 ' lr-AdamW/pg1‚0.00013349056815062824 ' lr-AdamW/pg2‚0.00013349056815062824 trainer/global_step‚3799 % _timestamp‚1.7884437929046328e+09iŠôÓÎèÈ _runtime‚3509.951868186 _step‚309 ' lr-AdamW/pg1‚0.00013349056815062824 ' lr-AdamW/pg2‚0.00013349056815062824 trainer/global_step‚3799 % _timestamp‚1.7884437929046328e+09µ`uº2é- _wandbruntime‚3509  _runtime‚3509GlDºò'êìO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82792287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3786465155584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150031257 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3598757870080 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39557311 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262009486937088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67238972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272946944962560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3599540995584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269348204262912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272947745258496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300113489756672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884437929053397e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269348204262912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82815397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67238970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269348187092480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20020696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20020181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216822784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3786657205760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265595256861422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150054369 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67238970 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296327049373696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚262009469897728 ŒõëñO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82815397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67238970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269348187092480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20020696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20020181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216822784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3786657205760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265595256861422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.alloc 0#áÝ ated‚ 150054369 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67238970 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296327049373696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚262009469897728 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82792287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3786465155584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150031257 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3598757870080 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39557311 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262009486937088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67238972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272946944962560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3599540995584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269348204262912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272947745258496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300113489756672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884437929053397e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269348204262912¶Z¥@)“ì‚(:ˆ ùåÔÆ›ËÈ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸öÚíVí‚(:Ë ùåÔ›‰øÉ disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.dm-1.out‚ 129.69921875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 25.017578125 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚ 8.870505e+06& disk.nvme0c0n1.out‚15374.025390625 disk.sda2.in‚0$ memory_percent‚95.78166007995605* disk./.usagePercent‚19.755929380260238! disk.dm-0.out‚15374.025390625 disk.drbd2265.out‚0* proc.memory.percent‚16.637349128723145 disk.sda3.out‚ 129.69921875% disk./.usageGB‚294.37762451171875 disk.sda.in‚0 disk.sda.out‚ 129.69921875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.86328125 disk.sda2.out‚0$ disk.nvme0n1.out‚15374.025390625 disk.drbd2080.in‚0 network.recv‚ 1.242518e+06 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.sda1.in‚0' proc.memory.availableMB‚ 691.1328125 cpu‚25.301390966065888ûF¾§2î- _wandbruntime‚3510  _runtime‚3510- ºeü&ïöM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273016504021504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59593009 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82836757 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150092769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82836757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296402731564032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3787456349184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585912679974 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269416818665984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67256010 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265662998606962 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296402706791424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82813621 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273017304330752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150069631 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39567306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884437937464862e+09 _runtime‚3510.794391653 _step‚311 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300190163140608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3599685355520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3599685355520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67256012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82813621 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20025703 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3600468494336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262076287686656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150069631 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59592499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300190369464320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67256010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20025196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273017304330752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150092769 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262076287686656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150069631 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59592499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300190369464320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67256010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20025196 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273017304330752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150092769 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150092769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82836757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296402731564032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3787456349184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585912679974 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269416818665984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67256010 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265662998606962 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296402706791424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82813621 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273017304330752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150069631 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39567306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884437937464862e+09 _runtime‚3510.794391653 _step‚311 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300190163140608·c6Œ„2ñ- _wandbruntime‚3510  _runtime‚3510Ç#x—[òÕ 1loss/SRqqt4/detection_loss‚1.2218129018564587 /loss/FRt1/assignment_loss‚4.230998758113746 1loss/SRqqt1/detection_loss‚2.2084520693336214 /loss/FBt2/detection_loss‚0.7734351419367108 trainer/global_step‚3799 /loss/FRt2/detection_loss‚1.7907648268155754 0loss/SRqqt3/detection_loss‚2.405815511090415 2loss/SRqqt3/assignment_loss‚2.1589144319295883 2loss/SRqqt4/assignment_loss‚1.9848825278813425 .loss/FRt4/detection_loss‚2.327369874891113 /loss/FBt3/detection_loss‚2.5915189544359842 epoch‚3 0loss/FBt4/assignment_loss‚0.5987306010045662 /loss/FBt1/detection_loss‚2.8206535279750824 0loss/FRt4/assignment_loss‚3.3850729699228324 0loss/FBt1/assignment_loss‚0.4209692190149023 1loss/SRqqt2/detection_loss‚1.7897931626066566 .loss/FBt4/detection_loss‚3.019399662812551 % _timestamp‚1.7884437937483346e+09 0loss/FRt2/assignment_loss‚2.5557108393737247 0loss/FBt3/assignment_loss‚0.6191177857330937 /loss/FRt3/detection_loss‚2.8624377462171737 _runtime‚3510.795469267 1loss/FBt2/assignment_loss‚0.28270577951796677 %loss/total_loss‚51.00649837993197 _step‚312 0loss/FRt3/assignment_loss‚4.0874929005099885 2loss/SRqqt1/assignment_loss‚1.8398489293952782 1loss/SRqqt2/assignment_loss‚2.147602956974879 /loss/FRt1/detection_loss‚2.8829973005887233 hQÑ?óÚ 1loss/FBt2/assignment_loss‚0.28270577951796677 %loss/total_loss‚51.00649837993197 _step‚312 0loss/FRt3/assignment_loss‚4.0874929005099885 2loss/SRqqt1/assignment_loss‚1.8398489293952782 1loss/SRqqt2/assignment_loss‚2.147602956974879 /loss/FRt1/detection_loss‚2.8829973005887233 1loss/SRqqt4/detection_loss‚1.2218129018564587 /loss/FRt1/assignment_loss‚4.230998758113746 1loss/SRqqt1/detection_loss‚2.2084520693336214 /loss/FBt2/detection_loss‚0.7734351419367108 trainer/global_step‚3799 /loss/FRt2/detection_loss‚1.7907648268155754 0loss/SRqqt3/detection_loss‚2.405815511090415 2loss/SRqqt3/assignment_loss‚2.1589144319295883 2loss/SRqqt4/assignment_loss‚1.9848825278813425 .loss/FRt4/detection_loss‚2.327369874891113 /loss/FBt3/detection_loss‚2.5915189544359842 epoch‚3 0loss/FBt4/assignment_loss‚0.5987306010045662 /loss/FBt1/detection_loss‚2.8206535279750824 0loss/FRt4/assignment_loss‚3.3850729699228324 0loss/FBt1/assignment_loss‚0.4209692190149023 1loss/SRqqt2/detection_loss‚1.7897931626066566 .loss/FBt4/detection_loss‚3.019399662812551 % _timestamp‚1.7884437937483346e+09 0loss/FRt2/assignment_loss‚2.5557108393737247 0loss/FBt3/assignment_loss‚0.6191177857330937 /loss/FRt3/detection_loss‚2.8624377462171737 &0äÚ!_runtime‚3510.795469267¸  žô‚(:“ ŸùåÔ¸ÖÇ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.367399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d´¶wRõ‚(:Ç ŸùåÔªŒæÉ% disk.nvme0c0n1.out‚15415.87109375 disk.nvme0n1.in‚0.078125* proc.memory.percent‚16.637349128723145 disk.sda3.in‚0 disk.sda3.out‚ 130.34765625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚15415.87109375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 130.34765625# disk.nvme0n1.out‚15415.87109375 disk.drbd2080.out‚0( proc.memory.availableMB‚ 691.26953125 disk.sda.out‚ 130.34765625 disk.drbd2266.out‚0 disk.rbd1.out‚ 25.1357421875 proc.cpu.threads‚17 network.sent‚ 8.967856e+06 disk.sda.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.249046e+06) disk./.usagePercent‚19.75699206753487 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.86328125 disk.nvme0c0n1.in‚0.078125# memory_percent‚95.7808256149292 cpu‚25.30040156193906% disk./.usageGB‚294.39345932006836ûag“ö‚(:ˆ ®ùåÔÜã È gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¥ùS÷‚(:È ®ùåÔ»²½É network.sent‚ 9.066392e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚15453.60546875 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.out‚ 130.75390625( proc.memory.availableMB‚ 691.26171875* proc.memory.percent‚16.637349128723145 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚15453.60546875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 130.75390625# disk.nvme0n1.out‚15453.60546875 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚25.29935306898179 disk.sda2.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.78087329864502 network.recv‚ 1.255426e+06 disk.sda1.out‚0 disk.sda3.out‚ 130.75390625# proc.memory.rssMB‚ 2725.86328125$ disk./.usageGB‚294.4167366027832* disk./.usagePercent‚19.758554225508803 disk.rbd1.in‚0 disk.rbd1.out‚ 25.2607421875 disk.sda2.out‚0 disk.drbd1716.in‚0‘š&•2ø- _wandbruntime‚3554  _runtime‚3554q8‚£Éùà ' lr-AdamW/pg1‚0.00013272861305703792 ' lr-AdamW/pg2‚0.00013272861305703792 trainer/global_step‚3849 % _timestamp‚1.7884438379029524e+09 _runtime‚3554.950192647 _step‚313_¼gÎúÈ ' lr-AdamW/pg1‚0.00013272861305703792 ' lr-AdamW/pg2‚0.00013272861305703792 trainer/global_step‚3849 % _timestamp‚1.7884438379029524e+09 _runtime‚3554.950192647 _step‚313¹åí¿ß2û- _wandbruntime‚3554  _runtime‚3554Ãû”ò'üìO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272779765767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276424912323584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276424912323584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68090972 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3645929681408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268982357719230 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151976421 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40056911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303946515987968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3554.951578122 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3645929681408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68090972 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303946730713600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 trainer/global_step‚3849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68090970 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40056914 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3631981335742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276425712619520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268981559479106 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268981559479106 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276424912323584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276424912323584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68090972 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3645929681408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268982357719230 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151976421 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40056911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303946515987968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3554.951578122 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3645929681408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68090972 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303946730713600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 trainer/global_step‚3849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68090970 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40056914 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3631981335742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276425712619520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ºv=+Íþ‚(:’ ½ùåÔ¾ãÑÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.895799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"In^ÿ‚(:Ó ½ùåÔÍòŽË cpu‚25.298284359033815 disk.sda3.in‚0$ memory_percent‚95.78235149383545 network.sent‚ 9.071577e+06 network.recv‚ 1.259128e+06 disk.drbd2080.in‚0 disk.rbd1.out‚ 25.3232421875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.637349128723145 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda3.out‚ 131.39453125( proc.memory.availableMB‚ 691.01953125 disk.sda.out‚ 131.39453125% disk./.usageGB‚294.42390060424805 disk.dm-1.out‚ 131.39453125& disk.nvme0n1.out‚15496.82568359375# disk.dm-0.out‚15496.82568359375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125* disk./.usagePercent‚19.759035007658106 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15496.82568359375# proc.memory.rssMB‚ 2725.86328125r Íf2€- _wandbruntime‚3555  _runtime‚3555´e…öM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276495271691776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272848397341184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83883673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304023393537024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3836552456192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 _step‚315 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151991683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20274304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265417177133056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83906809 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚3849 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276495271691776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304023180921856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3646857180160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300186841080832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40066903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269049301224438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3632124091382 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187842560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265417194172416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3555.788465588 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60341207 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272848414511616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3646074041344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20274833 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3632905292354 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276494471382528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151991683 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68108012 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68108010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3646857180160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60341739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83906809 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300186816308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3646074041344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68108012 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152014821 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152014821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3836364613632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83883673 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68108010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376yKx'‚ûM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60341207 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272848414511616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3646074041344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20274833 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3632905292354 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276494471382528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151991683 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68108012 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68108010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3646857180160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60341739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83906809 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300186816308224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3646074041344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68108012 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152014821 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152014821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3836364613632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83883673 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68108010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276495271691776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272848397341184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83883673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304023393537024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3836552456192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 _step‚315 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151991683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20274304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265417177133056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83906809 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 trainer/global_step‚3849 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276495271691776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304023180921856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3646857180160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300186841080832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40066903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269049301224438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3632124091382 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187842560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265417194172416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3555.788465588»~T,2ƒ- _wandbruntime‚3555  _runtime‚3555H¤WêX„Ò trainer/global_step‚3849 1loss/SRqqt2/assignment_loss‚1.368647444012918 0loss/FBt4/assignment_loss‚0.2230064157100225 /loss/FRt4/detection_loss‚2.2682146430015564 .loss/FBt2/detection_loss‚0.619082104577596 1loss/SRqqt1/detection_loss‚2.3233050384172578 0loss/SRqqt2/detection_loss‚1.528900367648978 1loss/SRqqt4/detection_loss‚1.5073828970990986 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9582769971296012 % _timestamp‚1.7884438387425482e+09 0loss/FRt4/assignment_loss‚3.9205008688962684 0loss/FBt1/assignment_loss‚0.7993602131803831 _step‚316 2loss/SRqqt3/assignment_loss‚1.7791283920858845 1loss/FBt3/assignment_loss‚0.41469856877537337 .loss/FRt3/detection_loss‚2.318275605854781 _runtime‚3555.790104461 /loss/FRt3/assignment_loss‚3.578952821700469 2loss/SRqqt1/assignment_loss‚1.8051231259979852 0loss/FBt2/assignment_loss‚0.1598807791389799 /loss/FRt1/detection_loss‚2.4265414588153362 /loss/FRt2/detection_loss‚1.8917143397309162 1loss/SRqqt3/detection_loss‚2.6931464599840567 /loss/FBt4/detection_loss‚3.0331133062189277 %loss/total_loss‚50.25824644225663 /loss/FRt1/assignment_loss‚4.750196203589439 0loss/FRt2/assignment_loss‚2.6895499693022833 .loss/FBt1/detection_loss‚3.993029773235321 .loss/FBt3/detection_loss‚2.2082186481531934ôñ]…× 2loss/SRqqt3/assignment_loss‚1.7791283920858845 1loss/FBt3/assignment_loss‚0.41469856877537337 .loss/FRt3/detection_loss‚2.318275605854781 _runtime‚3555.790104461 /loss/FRt3/assignment_loss‚3.578952821700469 2loss/SRqqt1/assignment_loss‚1.8051231259979852 0loss/FBt2/assignment_loss‚0.1598807791389799 /loss/FRt1/detection_loss‚2.4265414588153362 /loss/FRt2/detection_loss‚1.8917143397309162 1loss/SRqqt3/detection_loss‚2.6931464599840567 /loss/FBt4/detection_loss‚3.0331133062189277 %loss/total_loss‚50.25824644225663 /loss/FRt1/assignment_loss‚4.750196203589439 0loss/FRt2/assignment_loss‚2.6895499693022833 .loss/FBt1/detection_loss‚3.993029773235321 .loss/FBt3/detection_loss‚2.208218648153193 trainer/global_step‚3849 1loss/SRqqt2/assignment_loss‚1.368647444012918 0loss/FBt4/assignment_loss‚0.2230064157100225 /loss/FRt4/detection_loss‚2.2682146430015564 .loss/FBt2/detection_loss‚0.619082104577596 1loss/SRqqt1/detection_loss‚2.3233050384172578 0loss/SRqqt2/detection_loss‚1.528900367648978 1loss/SRqqt4/detection_loss‚1.5073828970990986 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9582769971296012 % _timestamp‚1.7884438387425482e+09 0loss/FRt4/assignment_loss‚3.9205008688962684 0loss/FBt1/assignment_loss‚0.7993602131803831 _step‚316¼D7­àž†‚(:“ ÌùåÔ´²ÅÇ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚136.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.244799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œžøxR‡‚(:Ç ÌùåÔ›ÙÝÉ disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125* disk./.usagePercent‚19.760111263332053 disk.dm-1.out‚ 131.76171875 disk.drbd1716.out‚0# disk.nvme0n1.out‚15536.23046875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 network.sent‚ 9.169242e+06 disk.dm-0.out‚15536.23046875 disk.sda.out‚ 131.76171875 disk.sda3.out‚ 131.76171875% proc.memory.availableMB‚ 690.96875 proc.cpu.threads‚17 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2725.86328125 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 disk.sda2.in‚0 cpu‚25.297199596916705 network.recv‚ 1.265862e+06 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚95.78266143798828* proc.memory.percent‚16.637349128723145% disk./.usageGB‚294.43993759155273 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 25.3232421875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚15536.23046875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0±ý𻓈‚(:ˆ ÛùåÔñ¯ØÇ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²rÇT‰‚(:É ÛùåÔã¥ÌÉ disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ memory_percent‚95.78192234039307 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.637349128723145 disk.sda3.in‚0 disk.dm-1.out‚ 132.52734375" disk.dm-0.out‚15581.8994140625 disk.rbd1.out‚ 25.41796875# proc.memory.rssMB‚ 2725.86328125 cpu‚25.29624398532155 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda.out‚ 132.52734375 disk.sda3.out‚ 132.52734375% disk.nvme0n1.out‚15581.8994140625 disk.drbd2266.in‚0( proc.memory.availableMB‚ 691.08984375 proc.cpu.threads‚17) disk./.usagePercent‚19.76194274123201 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 1.27869e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚0.078125 network.sent‚ 9.270902e+06# disk./.usageGB‚294.467227935791' disk.nvme0c0n1.out‚15581.8994140625 ÕC2Š- _wandbruntime‚3599  _runtime‚3599‰[WWÉ‹Ã ' lr-AdamW/pg1‚0.00013195504838525927 ' lr-AdamW/pg2‚0.00013195504838525927 trainer/global_step‚3899 % _timestamp‚1.7884438822115717e+09 _runtime‚3599.258812389 _step‚317g2–—ÎŒÈ _step‚317 ' lr-AdamW/pg1‚0.00013195504838525927 ' lr-AdamW/pg2‚0.00013195504838525927 trainer/global_step‚3899 % _timestamp‚1.7884438822115717e+09 _runtime‚3599.258812389½ñοY2- _wandbruntime‚3599  _runtime‚3599|\©ò'ŽìO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307779694581248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276211361613312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276211344442880 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84955501 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279902879684608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884438822123218e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303895243634688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3678192747150 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276211361613312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20518246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84932391 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68942972 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84932391 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268691265829888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272369458577038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268691248790528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84955501 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153898473 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61075286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3692318367232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3884232026624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153898473 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61075286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3692318367232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3884232026624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279902879684608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884438822123218e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303895243634688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3678192747150 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276211361613312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20518246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84932391 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68942972 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84932391 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268691265829888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272369458577038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268691248790528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84955501 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153913735 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692462727168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3693245865984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279972438743552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221003776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276279976016384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84953725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3885198107648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3678335502790 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3679116703762 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153913735 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303970925825024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 _step‚319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692462727168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84976861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20523153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276279993186816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84976861 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 trainer/global_step‚3899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153936873 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68960012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279973239052800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84953725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279973239052800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153936873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61089656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307856344936448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268758066579456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68960012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307856123932672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272437200322578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489&5ð'”úM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276279976016384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84953725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3885198107648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3678335502790 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3679116703762 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153913735 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303970925825024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 _step‚319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692462727168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84976861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20523153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276279993186816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84976861 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 trainer/global_step‚3899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153936873 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68960012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279973239052800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84953725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279973239052800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196231168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153936873 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61089656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307856344936448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268758066579456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68960012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307856123932672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272437200322578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68960010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153913735 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692462727168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3693245865984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279972438743552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221003776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283380847045632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283381647341568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933201549312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272032138236928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3738707053056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272032155276288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86025553 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283380847045632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 _runtime‚3643.681804903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3723622957794 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3737923927552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283381647341568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155820525 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279642940288512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86025553 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69794970 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311612356359168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155797413 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279642940288512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20768463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198341632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69794972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279642923118080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933003207680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275756559434846 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69794970 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3738707053056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41056111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20767933 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275755761194722 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279642940288512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20768463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198341632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69794972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279642923118080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933003207680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275756559434846 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69794970 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3738707053056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41056111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20767933 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275755761194722 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283380847045632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283381647341568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933201549312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272032138236928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3738707053056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272032155276288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264¸Eæ2¢- _wandbruntime‚3644  _runtime‚3644É…÷&£öM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307755035341824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279711554691584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283451206413824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69812012 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275823502940054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155858925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155835787 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307755060114432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3724546914198 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155835787 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69812010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41066103 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69812010 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275824301180386 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86023777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69812012 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86023777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41066106 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3933987796480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚3949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279711554691584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86046913 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311689023138304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61838944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283451206413824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3934173541888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20772841 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155858925 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3739634551808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279711571862016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272098956025856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.788443927447676˜I2e+09þF¸ˆ'¤ûM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275823502940054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155858925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155835787 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307755060114432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3724546914198 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155835787 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69812010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41066103 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69812010 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275824301180386 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86023777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69812012 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86023777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41066106 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3933987796480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚3949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279711554691584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86046913 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311689023138304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61838944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283451206413824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3934173541888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20772841 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155858925 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3739634551808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279711571862016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272098956025856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884439274476762e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307755035341824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279711554691584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283451206413824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69812012 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096ÃC„÷ü2¥- _wandbruntime‚3644  _runtime‚3644¸»ãÅW¦Ñ 0loss/FBt3/assignment_loss‚0.4495190158486366 _step‚324 0loss/FBt1/assignment_loss‚0.7178260527551175 0loss/FBt4/assignment_loss‚0.3886626649343978 1loss/SRqqt3/detection_loss‚2.4729992202350073 /loss/FBt3/detection_loss‚3.0264563858509064 1loss/SRqqt4/detection_loss‚1.4835098197926646 %loss/total_loss‚50.72970822751216 epoch‚3 trainer/global_step‚3949 /loss/FRt3/assignment_loss‚3.893631070000785 /loss/FRt4/assignment_loss‚2.935596681776501 1loss/SRqqt3/assignment_loss‚2.224021466714995 2loss/SRqqt4/assignment_loss‚2.0135717934564403 .loss/FRt2/assignment_loss‚3.45011568069458 1loss/SRqqt1/assignment_loss‚2.148177846074104 /loss/FRt4/detection_loss‚2.4288632018225536 /loss/FBt2/detection_loss‚0.7587116859663314 /loss/FBt4/detection_loss‚2.9446814258893332 % _timestamp‚1.7884439274494169e+09 /loss/FRt3/detection_loss‚2.8708631685801915 1loss/SRqqt2/detection_loss‚1.6586950838565826 .loss/FRt1/detection_loss‚2.498477900444075 .loss/FBt1/detection_loss‚3.065778040885925 2loss/SRqqt2/assignment_loss‚1.1171527044847607 1loss/FBt2/assignment_loss‚0.15457481051022587 _runtime‚3644.496605824 0loss/FRt1/assignment_loss‚3.9186535325161245 0loss/SRqqt1/detection_loss‚1.935580222606659 /loss/FRt2/detection_loss‚2.1735887518152595ÖÝýÐ\§Ö 1loss/SRqqt3/assignment_loss‚2.224021466714995 2loss/SRqqt4/assignment_loss‚2.0135717934564403 .loss/FRt2/assignment_loss‚3.45011568069458 1loss/SRqqt1/assignment_loss‚2.148177846074104 /loss/FRt4/detection_loss‚2.4288632018225536 /loss/FBt2/detection_loss‚0.7587116859663314 /loss/FBt4/detection_loss‚2.9446814258893332 % _timestamp‚1.7884439274494169e+09 /loss/FRt3/detection_loss‚2.8708631685801915 1loss/SRqqt2/detection_loss‚1.6586950838565826 .loss/FRt1/detection_loss‚2.498477900444075 .loss/FBt1/detection_loss‚3.065778040885925 2loss/SRqqt2/assignment_loss‚1.1171527044847607 1loss/FBt2/assignment_loss‚0.15457481051022587 _runtime‚3644.496605824 0loss/FRt1/assignment_loss‚3.9186535325161245 0loss/SRqqt1/detection_loss‚1.935580222606659 /loss/FRt2/detection_loss‚2.1735887518152595 0loss/FBt3/assignment_loss‚0.4495190158486366 _step‚324 0loss/FBt1/assignment_loss‚0.7178260527551175 0loss/FBt4/assignment_loss‚0.3886626649343978 1loss/SRqqt3/detection_loss‚2.4729992202350073 /loss/FBt3/detection_loss‚3.0264563858509064 1loss/SRqqt4/detection_loss‚1.4835098197926646 %loss/total_loss‚50.72970822751216 epoch‚3 trainer/global_step‚3949 /loss/FRt3/assignment_loss‚3.893631070000785 /loss/FRt4/assignment_loss‚2.935596681776501ľHzù‘¨‚(:† —úåÔ•ùÖÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÔèÝZ©‚(:Ï —úåÔòÂÍÊ disk.sda2.out‚0 disk.dm-1.out‚ 134.50390625& disk.nvme0n1.out‚15739.66748046875 disk.sda1.in‚0 network.sent‚ 9.481295e+06 network.recv‚ 1.305866e+06 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.in‚0* proc.memory.percent‚16.637349128723145 disk.rbd1.out‚ 25.6376953125 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0$ memory_percent‚95.78466415405273 disk.rbd1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚15739.66748046875 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0) disk./.usagePercent‚19.76419048738264 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 134.50390625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 134.50390625 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 disk.drbd2080.out‚0& proc.memory.availableMB‚ 690.640625# proc.memory.rssMB‚ 2725.86328125 cpu‚25.292205147719326$ disk./.usageGB‚294.5007209777832 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15739.66748046875¿#-žª‚(:“ ¦úåÔÍóçÇ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.504999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦o£Y«‚(:Î ¦úåÔÁƒÛÉ* disk./.usagePercent‚19.767298418848313 disk.drbd2265.out‚0 disk.rbd1.out‚ 25.7626953125 disk.nvme0c0n1.in‚0.078125 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 135.1640625( proc.memory.availableMB‚ 690.63671875 network.sent‚ 9.590343e+06 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 135.1640625 proc.cpu.threads‚17 network.recv‚ 1.312363e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.637349128723145 disk.rbd0.in‚0 disk.sda.out‚ 135.1640625& disk.nvme0n1.out‚15763.26904296875# disk.dm-0.out‚15763.26904296875 cpu‚25.29123207709029$ disk./.usageGB‚294.5470314025879 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.078125$ memory_percent‚95.78468799591064 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125# proc.memory.rssMB‚ 2725.86328125 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚15763.26904296875ç:ŒÇž¬‚(:“ µúåÔÓ³¿Ç gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸" tO­‚(:Ä µúåÔ¤úøÊ disk.dm-1.out‚ 135.84375 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0$ memory_percent‚95.78664302825928( proc.memory.availableMB‚ 690.31640625* proc.memory.percent‚16.637349128723145 disk.drbd2266.out‚0 network.recv‚ 1.316002e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.rbd0.out‚0 network.sent‚ 9.595504e+06$ disk./.usageGB‚294.6125717163086 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 25.9111328125 proc.cpu.threads‚17% disk.nvme0n1.out‚15808.8701171875 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2725.86328125 cpu‚25.2903044286776 disk.sda.out‚ 135.84375 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚15808.8701171875 disk.sda3.in‚0 disk.sda3.out‚ 135.84375" disk.dm-0.out‚15808.8701171875 disk.drbd2080.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚19.771696884294098˜À—º2®- _wandbruntime‚3687  _runtime‚3687XÖÅǯÁ trainer/global_step‚3999 $ _timestamp‚1.788443970007185e+09 _runtime‚ 3687.05441404 _step‚325 ' lr-AdamW/pg1‚0.00013037373753183254 ' lr-AdamW/pg2‚0.00013037373753183254ø@ëíÌ°Æ ' lr-AdamW/pg1‚0.00013037373753183254 ' lr-AdamW/pg2‚0.00013037373753183254 trainer/global_step‚3999 $ _timestamp‚1.788443970007185e+09 _runtime‚ 3687.05441404 _step‚325Å°¥™2±- _wandbruntime‚3687  _runtime‚3687y(Räò'²ìO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70646970 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286858814406656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315445588972544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87072495 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286859614702592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286859614702592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3784312613376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70646970 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279143660292654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3785095738880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157742577 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3982101531648 _step‚326 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41555711 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275373027683328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3981913675776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157719465 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87072495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286858814406656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21017628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315445376344064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3770615569966 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚3687.055363552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283074501793280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21018165 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311463462668288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283074501793280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87095605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62573879 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70646972 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283074518963712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283074518963712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769834369202 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 ×*æ³ñO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283074518963712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283074518963712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769834369202 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70646970 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286858814406656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315445588972544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87072495 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286859614702592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286859614702592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3784312613376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70646970 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279143660292654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytÓwes.large_pool.peak‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3785095738880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157742577 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3982101531648 _step‚326 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41555711 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275373027683328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3981913675776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157719465 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87072495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286858814406656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21017628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315445376344064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3770615569966 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚3687.055363552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283074501793280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21018165 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311463462668288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283074501793280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87095605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62573879 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70646972Æœ)F2´- _wandbruntime‚3687  _runtime‚3687Ø;Ãü&µöM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283143150537216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157757839 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 _step‚327 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286928373465600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41565703 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157757839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311539169631232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3982893039104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275439862511616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286929173774848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87093829 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3785240098816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3785240098816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770758325606 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87116965 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87116965 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41565706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315522037897728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3983078784512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279210603797862 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3771539526578 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3786023237632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚3999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157780977 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283143133366784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62588344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283143150537216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _runtime‚3687.862969188 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70664010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87093829 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157780977 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315522248415744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70664010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279211402038194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70664012 ËÕ9'¶ûM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770758325606 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87116965 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87116965 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41565706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315522037897728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3983078784512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279210603797862 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3771539526578 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3786023237632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚3999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157780977 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283143133366784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62588344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283143150537216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _runtime‚3687.862969188 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70664010 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87093829 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157780977 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315522248415744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70664010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279211402038194 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70664012 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283143150537216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157757839 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 _step‚327 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286928373465600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41565703 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157757839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311539169631232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3982893039104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275439862511616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286929173774848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87093829 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3785240098816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3785240098816Çè^— 2·- _wandbruntime‚3687  _runtime‚3687iïaóT¸Î 1loss/SRqqt3/assignment_loss‚1.977944315626071 .loss/FRt1/detection_loss‚2.423753976076841 0loss/SRqqt1/detection_loss‚2.105587111833768 /loss/FRt3/assignment_loss‚4.091635058323543 .loss/FRt3/detection_loss‚2.782193342844645 1loss/SRqqt3/detection_loss‚2.4217715687476673 epoch‚3 /loss/FRt1/assignment_loss‚4.007788729667664 2loss/SRqqt4/assignment_loss‚1.9675977782524647 0loss/FBt3/assignment_loss‚0.5504791565799678 1loss/SRqqt4/detection_loss‚1.3104105387279328 /loss/FRt2/assignment_loss‚2.964285833305783 1loss/FBt1/assignment_loss‚0.36618858261208515 /loss/FRt2/detection_loss‚1.9922450956371096 /loss/FBt2/detection_loss‚0.8934312921066858 _runtime‚ 3687.86449084 trainer/global_step‚3999 # _timestamp‚1.78844397081734e+09 2loss/SRqqt2/assignment_loss‚1.9099048674106598 0loss/SRqqt2/detection_loss‚1.690725657769612 _step‚328 %loss/total_loss‚51.84865499870436 0loss/FRt4/assignment_loss‚3.6048517421234485 1loss/FBt2/assignment_loss‚0.26827676557762664 0loss/FBt4/assignment_loss‚0.5249339141576327 .loss/FRt4/detection_loss‚2.442934516557427 .loss/FBt1/detection_loss‚3.232561635971069 /loss/FBt4/detection_loss‚3.4629302450588773 2loss/SRqqt1/assignment_loss‚1.8846722130592053 /loss/FBt3/detection_loss‚2.9715510606765747œ¿õHY¹Ó %loss/total_loss‚51.84865499870436 0loss/FRt4/assignment_loss‚3.6048517421234485 1loss/FBt2/assignment_loss‚0.26827676557762664 0loss/FBt4/assignment_loss‚0.5249339141576327 .loss/FRt4/detection_loss‚2.442934516557427 .loss/FBt1/detection_loss‚3.232561635971069 /loss/FBt4/detection_loss‚3.4629302450588773 2loss/SRqqt1/assignment_loss‚1.8846722130592053 /loss/FBt3/detection_loss‚2.9715510606765747 1loss/SRqqt3/assignment_loss‚1.977944315626071 .loss/FRt1/detection_loss‚2.423753976076841 0loss/SRqqt1/detection_loss‚2.105587111833768 /loss/FRt3/assignment_loss‚4.091635058323543 .loss/FRt3/detection_loss‚2.782193342844645 1loss/SRqqt3/detection_loss‚2.4217715687476673 epoch‚3 /loss/FRt1/assignment_loss‚4.007788729667664 2loss/SRqqt4/assignment_loss‚1.9675977782524647 0loss/FBt3/assignment_loss‚0.5504791565799678 1loss/SRqqt4/detection_loss‚1.3104105387279328 /loss/FRt2/assignment_loss‚2.964285833305783 1loss/FBt1/assignment_loss‚0.36618858261208515 /loss/FRt2/detection_loss‚1.9922450956371096 /loss/FBt2/detection_loss‚0.8934312921066858 _runtime‚ 3687.86449084 trainer/global_step‚3999 # _timestamp‚1.78844397081734e+09 2loss/SRqqt2/assignment_loss‚1.9099048674106598 0loss/SRqqt2/detection_loss‚1.690725657769612 _step‚328È2ñ6V“º‚(:ˆ ÄúåÔ¶ÈóÉ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a¯ØZ»‚(:Ï ÄúåÔ§ºÂÊ* disk./.usagePercent‚19.772361479852986 disk.rbd1.out‚ 25.9111328125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.recv‚ 1.320939e+06 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 136.37890625# proc.memory.rssMB‚ 2725.86328125 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125" disk.dm-0.out‚15844.7060546875 disk.drbd2266.in‚0( proc.memory.availableMB‚ 690.54296875 cpu‚25.28934030791502' disk.nvme0c0n1.out‚15844.7060546875 disk.drbd2266.out‚0$ memory_percent‚95.78526020050049 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.637349128723145 disk.sda1.out‚0 disk.sda.out‚ 136.37890625 disk.nvme0n1.in‚0.078125 proc.cpu.threads‚17 network.sent‚ 9.602232e+06% disk./.usageGB‚294.62247467041016 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 136.37890625% disk.nvme0n1.out‚15844.70605468755¤¦“¼‚(:ˆ ÓúåÔ·ìÝÇ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âß$èR½‚(:Ç ÓúåÔ¤ÜÈÉ* disk./.usagePercent‚19.775627623974824 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.86328125 disk.drbd1716.out‚0 network.recv‚ 1.339986e+06 disk.sda3.out‚ 136.90625 disk.drbd2080.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15893.78173828125 cpu‚25.288324602011308# disk./.usageGB‚294.671142578125 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 25.943359375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚95.78702449798584( proc.memory.availableMB‚ 690.25390625* proc.memory.percent‚16.637349128723145 disk.dm-1.out‚ 136.90625 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 network.sent‚ 9.717149e+06 disk.sda.out‚ 136.90625 disk.nvme0n1.in‚0.078125& disk.nvme0n1.out‚15893.78173828125# disk.dm-0.out‚15893.78173828125 disk.drbd2266.out‚0 disk.drbd2266.in‚0Þ^%„“¾‚(:ˆ âúåÔÇ gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÕÊ«!¿‚(:Í âúåÔÆαÉ* disk./.usagePercent‚19.771228390506437$ disk./.usageGB‚294.6055908203125 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 690.0078125 network.recv‚ 1.351821e+06 disk.dm-1.out‚ 137.61328125 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚15932.9326171875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚15932.9326171875% disk.nvme0n1.out‚15932.9326171875 disk.rbd0.out‚0 cpu‚25.287306179538657 disk.sda3.in‚0 disk.sda3.out‚ 137.61328125 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.86328125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2265.out‚hŽ70$ memory_percent‚95.78852653503418 disk.rbd1.out‚ 26.248046875 disk.sda.out‚ 137.61328125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.in‚0* proc.memory.percent‚16.637349128723145 proc.cpu.threads‚17 network.sent‚ 9.727434e+06 disk.dm-1.in‚0{éò¿2À- _wandbruntime‚3730  _runtime‚3730R–³ÛÇÁÁ _step‚329 & lr-AdamW/pg1‚0.0001295663208929962 & lr-AdamW/pg2‚0.0001295663208929962 trainer/global_step‚4049 % _timestamp‚1.7884440131217308e+09 _runtime‚3730.168975026hd~ÌÂÆ & lr-AdamW/pg2‚0.0001295663208929962 trainer/global_step‚4049 % _timestamp‚1.7884440131217308e+09 _runtime‚3730.168975026 _step‚329 & lr-AdamW/pg1‚0.0001295663208929962Éžkõ2Ã- _wandbruntime‚3730  _runtime‚3730°·P ò'ÄìO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315247572185088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3816826981374 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63324680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3831484424704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3830701299200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319278552507392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21269366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21268840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71498972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _step‚330 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71498970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282530761150462 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159664629 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290336781767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286506097638912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42055311 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884440131224601e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3831484424704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290336781767680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198341632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010Ð;b°÷'ÅñO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71498972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 _step‚330 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71498970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282530761150462 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159664629 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290336781767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286506097638912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42055311 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884440131224601e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3831484424704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290336781767680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198341632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315247572185088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3816826981374 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63324680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3831484424704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3830701299200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319278552507392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21269366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21268840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489ÊC  82Æ- _wandbruntime‚3731  _runtime‚3731­N¦‘ü&ÇöM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88187017 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71516010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290407141135872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3832411923456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71516012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278780751958016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4031918344704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159679891 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290407141135872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159703029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42065303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71516012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286574712041984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63339569 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88187017 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4031720016384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159703029 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278780734918656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88163881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286574729212416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286574729212416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282598502896002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831628784640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42065306 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3831628784640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚3731.001687413 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290406340826624 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63339037 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3832411923456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71516010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3816969737014 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290407141135872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3832411923456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71516012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278780751958016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4031918344704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159679891 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290407141135872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159703029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42065303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71516012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286574712041984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63339569 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88187017 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4031720016384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159703029 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278780734918656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88163881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286574729212416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286574729212416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282598502896002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831628784640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42065306 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3831628784640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _runtime‚3731.001687413 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290406340826624 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63339037 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3832411923456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71516010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3816969737014 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0˦ -ƒ2É- _wandbruntime‚3731  _runtime‚3731×Ì=7ZÊÔ 0loss/FBt2/assignment_loss‚0.2518752602764505 /loss/FRt4/detection_loss‚2.1523684165694497 0loss/FRt3/assignment_loss‚3.7768409848213196 0loss/FBt4/assignment_loss‚1.3204009905457497 0loss/SRqqt2/detection_loss‚2.085206397316035 1loss/SRqqt4/detection_loss‚1.2841581776738167 _step‚332 2loss/SRqqt1/assignment_loss‚1.8421040431774678 /loss/FBt1/detection_loss‚3.8845936059951782 _runtime‚3731.002738106 .loss/FBt3/detection_loss‚2.232442357960869 epoch‚3 $ _timestamp‚1.788444013955551e+09 2loss/SRqqt2/assignment_loss‚2.1895952575347004 0loss/FBt3/assignment_loss‚0.5860627077474205 /loss/FBt2/detection_loss‚0.6521980641785942 trainer/global_step‚4049 1loss/FBt1/assignment_loss‚0.22150065501530966 /loss/FRt3/detection_loss‚2.7557686679065228 1loss/SRqqt1/detection_loss‚1.9696216221819534 0loss/SRqqt3/detection_loss‚2.710255659543551 /loss/FBt4/detection_loss‚4.9604723155498505 0loss/FRt4/assignment_loss‚2.9601830742575905 1loss/SRqqt3/assignment_loss‚2.611238642380788 2loss/SRqqt4/assignment_loss‚1.7266545609652417 /loss/FRt1/detection_loss‚2.5317153597986977 /loss/FRt1/assignment_loss‚4.093141997969428 0loss/FRt2/assignment_loss‚3.6905470645427703 /loss/FRt2/detection_loss‚1.8724061471223832 %loss/total_loss‚54.36135203103114”ö_ËÙ 0loss/FRt4/assignment_loss‚2.9601830742575905 1loss/SRqqt3/assignment_loss‚2.611238642380788 2loss/SRqqt4/assignment_loss‚1.7266545609652417 /loss/FRt1/detection_loss‚2.5317153597986977 /loss/FRt1/assignment_loss‚4.093141997969428 0loss/FRt2/assignment_loss‚3.6905470645427703 /loss/FRt2/detection_loss‚1.8724061471223832 %loss/total_loss‚54.36135203103114 0loss/FBt2/assignment_loss‚0.2518752602764505 /loss/FRt4/detection_loss‚2.1523684165694497 0loss/FRt3/assignment_loss‚3.7768409848213196 0loss/FBt4/assignment_loss‚1.3204009905457497 0loss/SRqqt2/detection_loss‚2.085206397316035 1loss/SRqqt4/detection_loss‚1.2841581776738167 _step‚332 2loss/SRqqt1/assignment_loss‚1.8421040431774678 /loss/FBt1/detection_loss‚3.8845936059951782 _runtime‚3731.002738106 .loss/FBt3/detection_loss‚2.232442357960869 epoch‚3 $ _timestamp‚1.788444013955551e+09 2loss/SRqqt2/assignment_loss‚2.1895952575347004 0loss/FBt3/assignment_loss‚0.5860627077474205 /loss/FBt2/detection_loss‚0.6521980641785942 trainer/global_step‚4049 1loss/FBt1/assignment_loss‚0.22150065501530966 /loss/FRt3/detection_loss‚2.7557686679065228 1loss/SRqqt1/detection_loss‚1.9696216221819534 0loss/SRqqt3/detection_loss‚2.710255659543551 /loss/FBt4/detection_loss‚4.9604723155498505ÌÎ4“Ì‚(:ˆ ñúåÔÝÆ»Ç gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.7944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¨wMÍ‚(: ñúåÔ­‘ªÊ disk.rbd1.out‚ 26.662109375 disk.sda1.out‚0 disk.sda3.out‚ 138.2421875 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 138.2421875 disk.drbd1716.out‚0# disk.nvme0n1.out‚15979.71484375 cpu‚25.28634669642937 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚15979.71484375" proc.memory.rssMB‚ 2725.8671875% disk.nvme0c0n1.out‚15979.71484375 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.637372970581055$ disk./.usageGB‚294.7250633239746 disk.rbd1.in‚0 network.recv‚ 1.355526e+06 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda.out‚ 138.2421875 disk.nvme0n1.in‚0.078125 network.sent‚ 9.732719e+06 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.drbd1716.in‚0$ memory_percent‚95.78745365142822( proc.memory.availableMB‚ 690.18359375) disk./.usagePercent‚19.77924629047131 disk.drbd2080.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0Õ†G“΂(:ˆ €ûåÔÜï¨Ç gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PoF FÏ‚(:» €ûåÔ¨ÜÌÉ disk.sda2.in‚0 disk.sda3.out‚139.0625 disk.rbd1.out‚ 26.662109375 disk.sda1.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2725.8671875* disk./.usagePercent‚19.785050493287606 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125# disk.nvme0n1.out‚16018.60546875 disk.rbd0.in‚0* proc.memory.percent‚16.637372970581055 disk.dm-1.out‚139.0625 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚16018.60546875 disk.rbd1.in‚0 disk.sda.out‚139.0625 network.recv‚ 1.374723e+06 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚16018.60546875$ memory_percent‚95.78919410705566 cpu‚25.285417884015008 network.sent‚ 9.847667e+06% disk./.usageGB‚294.81155014038086 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125' proc.memory.availableMB‚ 689.8984375Y­lV‘Ђ(:† ûåÔ±ðìÉ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cðm±?Ñ‚(:´ ûåÔþÉŒÌ disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.284451635093756 network.recv‚ 1.378992e+06 disk.nvme0n1.in‚0.078125" disk.nvme0n1.out‚ 16063.5546875 disk.drbd2265.out‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 690.40625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 16063.5546875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 26.724609375$ memory_percent‚95.78609466552734" proc.memory.rssMB‚ 2725.8671875* proc.memory.percent‚16.637372970581055 network.sent‚ 9.853451e+06 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 16063.5546875 disk.sda2.out‚0 disk.sda3.out‚139.6875 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0$ disk./.usageGB‚294.8512725830078 disk.dm-1.out‚139.6875 disk.sda.out‚139.6875 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0) disk./.usagePercent‚19.78771629974165®°¹¢2Ò- _wandbruntime‚3773  _runtime‚3773L´XÁÈÓ ' lr-AdamW/pg1‚0.00012874795369291874 ' lr-AdamW/pg2‚0.00012874795369291874 trainer/global_step‚4099 $ _timestamp‚1.788444056456462e+09 _runtime‚3773.503683012 _step‚333éãéæÍÔÇ ' lr-AdamW/pg1‚0.00012874795369291874 ' lr-AdamW/pg2‚0.00012874795369291874 trainer/global_step‚4099 $ _timestamp‚1.788444056456462e+09 _runtime‚3773.503683012 _step‚333ÍUqÓ¸2Õ- _wandbruntime‚3773  _runtime‚3773Ãb®æñ'ÖëO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293815549424640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161563569 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282054823615488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72350970 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89235709 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183661568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289937659143680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161586681 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289937676314112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21520421 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3877873110528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319031706474496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4079772049408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323111662185472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293814749128704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3863038392782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚4099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3877089985024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323111453751296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¥b¦×ðO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323111662185472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293814749128704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3863038392782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 trainer/global_step‚4099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3877089985024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323111453751296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293815549424640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161563569 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282054823615488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72350970 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89235709 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183661568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289937659143680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161586681 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289937676314112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21520421 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3877873110528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319031706474496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4079772049408Î綿¹2Ø- _wandbruntime‚3774  _runtime‚3774o‚¹¬ü&ÙöM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319107388664832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72368012 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3878017470464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323188104684032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293885108496896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64090267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161625081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64090796 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319107363892224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282121641404416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282121624365056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72368010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290006290717184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3878800609280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290006307887616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290006307887616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3878017470464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3863962349394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89233933 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 trainer/global_step‚4099 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3863181148422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72368012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293884308187648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323188327784960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚3774.322861223 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42564903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285984805513478 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚335 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89233933 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293885108496896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89257069 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161625081 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3878800609280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161601943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4080740791808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884440572749612e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161601943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21525364gfÅ'ÚûM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42564903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285984805513478 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚335 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89233933 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293885108496896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89257069 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161625081 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3878800609280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161601943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4080740791808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884440572749612e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161601943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21525364 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319107388664832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72368012 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3878017470464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323188104684032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293885108496896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64090267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198328320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161625081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64090796 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319107363892224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282121641404416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282121624365056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72368010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290006290717184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3878800609280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290006307887616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290006307887616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3878017470464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3863962349394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89233933 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 trainer/global_step‚4099 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3863181148422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72368012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293884308187648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323188327784960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚3774.322861223 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865Ï“Á&ó2Û- _wandbruntime‚3774  _runtime‚3774MLÖ1UÜÏ trainer/global_step‚4099 /loss/FBt4/detection_loss‚2.9693371984693737 .loss/FRt1/detection_loss‚2.719583137333393 /loss/FRt3/detection_loss‚2.7069205731153487 $loss/total_loss‚52.0872967541074 $ _timestamp‚1.788444057276707e+09 _runtime‚3774.323832686 2loss/SRqqt1/assignment_loss‚1.7350944208132255 2loss/SRqqt3/assignment_loss‚2.6012615174469023 /loss/FRt1/assignment_loss‚4.066176223754883 .loss/FRt4/detection_loss‚2.437642128765583 0loss/SRqqt1/detection_loss‚2.069028789127195 1loss/SRqqt2/detection_loss‚1.9117401549592614 1loss/SRqqt4/detection_loss‚1.3884803391191909 .loss/FBt1/detection_loss‚3.389452406338283 /loss/FRt3/assignment_loss‚4.418012228608132 1loss/SRqqt2/assignment_loss‚1.459820545394905 1loss/FBt1/assignment_loss‚0.18920085480515678 0loss/FBt3/assignment_loss‚0.3364717762958915 /loss/FRt2/detection_loss‚2.2784280821326233 /loss/FBt2/detection_loss‚0.6633461122862236 /loss/FBt3/detection_loss‚2.7847295999526978 epoch‚3 /loss/FRt4/assignment_loss‚3.447539100050926 2loss/SRqqt4/assignment_loss‚2.0561069476027645 1loss/FBt2/assignment_loss‚0.25119642926808394 _step‚336 0loss/FRt2/assignment_loss‚3.1916079961118244 /loss/FBt4/assignment_loss‚0.760295101616066 0loss/SRqqt3/detection_loss‚2.255825090739462Q¾R:ZÝÔ /loss/FRt1/assignment_loss‚4.066176223754883 .loss/FRt4/detection_loss‚2.437642128765583 0loss/SRqqt1/detection_loss‚2.069028789127195 1loss/SRqqt2/detection_loss‚1.9117401549592614 1loss/SRqqt4/detection_loss‚1.3884803391191909 .loss/FBt1/detection_loss‚3.389452406338283 /loss/FRt3/assignment_loss‚4.418012228608132 1loss/SRqqt2/assignment_loss‚1.459820545394905 1loss/FBt1/assignment_loss‚0.18920085480515678 0loss/FBt3/assignment_loss‚0.3364717762958915 /loss/FRt2/detection_loss‚2.2784280821326233 /loss/FBt2/detection_loss‚0.6633461122862236 /loss/FBt3/detection_loss‚2.7847295999526978 epoch‚3 /loss/FRt4/assignment_loss‚3.447539100050926 2loss/SRqqt4/assignment_loss‚2.0561069476027645 1loss/FBt2/assignment_loss‚0.25119642926808394 _step‚336 0loss/FRt2/assignment_loss‚3.1916079961118244 /loss/FBt4/assignment_loss‚0.760295101616066 0loss/SRqqt3/detection_loss‚2.255825090739462 trainer/global_step‚4099 /loss/FBt4/detection_loss‚2.9693371984693737 .loss/FRt1/detection_loss‚2.719583137333393 /loss/FRt3/detection_loss‚2.7069205731153487 $loss/total_loss‚52.0872967541074 $ _timestamp‚1.788444057276707e+09 _runtime‚3774.323832686 2loss/SRqqt1/assignment_loss‚1.7350944208132255 2loss/SRqqt3/assignment_loss‚2.6012615174469023ÐOÖHžÞ‚(:“ žûåÔ®á²È gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.294199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³)ÔxHß‚(:½ žûåÔÇéÁÊ! disk.nvme0n1.out‚ 16098.921875 disk.dm-0.out‚ 16098.921875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.recv‚ 1.396592e+06% disk./.usageGB‚294.92149353027344 disk.dm-1.in‚0 disk.rbd1.out‚ 26.849609375# memory_percent‚95.7895278930664* proc.memory.percent‚16.637372970581055 disk.sda2.in‚0 disk.sda3.out‚ 140.48828125 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125# disk.nvme0c0n1.out‚ 16098.921875" proc.memory.rssMB‚ 2725.8671875 cpu‚25.28354967614039 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125% proc.memory.availableMB‚ 689.84375 network.sent‚ 9.866579e+06 disk.dm-1.out‚ 140.48828125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda.out‚ 140.48828125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0* disk./.usagePercent‚19.792428886431946 disk.sda.in‚0”àŠÖ“à‚(:ˆ ­ûåÔòŒ¶Ç gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚190.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™˜X«Oá‚(:Ä ­ûåÔ–ãžÊ$ memory_percent‚95.78804969787598% disk./.usageGB‚295.07189559936523$ disk.nvme0c0n1.out‚ 16143.7734375 disk.sda.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.recv‚ 1.402023e+06 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.sda1.out‚0* disk./.usagePercent‚19.802522495484435 disk.dm-1.in‚0 disk.drbd1716.out‚0 cpu‚25.282571138655435 network.sent‚ 9.974248e+06 disk.sda3.in‚0 disk.dm-1.out‚ 140.99609375 disk.drbd2266.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 140.99609375* proc.memory.percent‚16.637372970581055 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.dm-0.out‚ 16143.7734375 disk.rbd1.in‚0' proc.memory.availableMB‚ 690.0859375" proc.memory.rssMB‚ 2725.8671875 disk.sda3.out‚ 140.99609375" disk.nvme0n1.out‚ 16143.7734375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 26.912109375 disk.rbd0.out‚0 disk.sda1.in‚0Oza˜žâ‚(:“ ¼ûåÔ‡ÐÊÇ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.017' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.203399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú¿¨›Jã‚(:¿ ¼ûåÔÆýáÉ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 141.765625 proc.cpu.threads‚17% disk./.usageGB‚295.16131591796875 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 141.765625 disk.drbd1716.out‚0 disk.dm-0.out‚ 16180.3515625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 cpu‚25.281668192006027* disk./.usagePercent‚19.808523568094564 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.078125$ disk.nvme0c0n1.out‚ 16180.3515625 disk.sda3.out‚ 141.765625' proc.memory.availableMB‚ 689.8359375 network.sent‚ 9.980213e+06 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0 disk.rbd1.out‚ 26.912109375# proc.memory.rssMB‚ 2725.87109375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚ 1.406296e+06$ memory_percent‚95.78957557678223* proc.memory.percent‚16.637396812438965" disk.nvme0n1.out‚ 16180.3515625 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0-å{i2ä- _wandbruntime‚3816  _runtime‚3816*\„lÈå _runtime‚3816.719449508 _step‚337 ' lr-AdamW/pg1‚0.00012791880647809815 ' lr-AdamW/pg2‚0.00012791880647809815 trainer/global_step‚4149 $ _timestamp‚1.788444099672257e+09‰’ qÍæÇ ' lr-AdamW/pg1‚0.00012791880647809815 ' lr-AdamW/pg2‚0.00012791880647809815 trainer/global_step‚4149 $ _timestamp‚1.788444099672257e+09 _runtime‚3816.719449508 _step‚337ÑY’â#2ç- _wandbruntime‚3816  _runtime‚3816g+‘yèëO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43054511 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297293516785664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 YEDeviceStatsMonitor.on_train_batch_start/8]V€x%requested_bytes.all.allocated‚289304962866078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90305761 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293369237818880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297292716489728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322815791218688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4128946881536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3924261796352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923478670848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326944548147200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 _runtime‚ 3816.72039328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 % _timestamp‚1.7884440996729631e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163508733 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90282651 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285395713061888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163485621 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73202970 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3924261796352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293369254989312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64826553 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297292716489728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21772042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3923478670848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73202972 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43054514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293369254989312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322815815991296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21772572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3908468603426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112”Cö'éðO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297292716489728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322815791218688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4128946881536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3924261796352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923478670848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326944548147200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 _runtime‚ 3816.72039328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 % _timestamp‚1.7884440996729631e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163508733 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90282651 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285395713061888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163485621 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73202970 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3924261796352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293369254989312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64826553 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297292716489728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21772042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3923478670848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73202972 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43054514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293369254989312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322815815991296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21772572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3908468603426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43054511 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297293516785664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚289304962866078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90305761 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293369237818880Ò mÎO2ê- _wandbruntime‚3817  _runtime‚3817:nø‰û&ëõM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73220012 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73220010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297363075857920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90327121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163547133 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129925188096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289371906371286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3909392559830 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293437869392384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163523995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90303985 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 _runtime‚3817.515940962 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293437886562816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3925189295104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297362275548672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3910173760802 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293437886562816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90303985 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297363075857920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21777557 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289372704611618 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285462530850816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163547133 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3925189295104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64841539 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3924406156288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322891498181632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322891473409024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285462513811456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73220010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚4149 F#'éB ìúM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289372704611618 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285462530850816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163547133 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3925189295104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64841539 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3924406156288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322891498181632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322891473409024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285462513811456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73220010 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 trainer/global_step‚4149 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163547133 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129925188096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289371906371286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3909392559830 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293437869392384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163523995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210518016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90303985 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 _runtime‚3817.515940962 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293437886562816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3925189295104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297362275548672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3910173760802 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293437886562816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90303985 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297363075857920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21777557Óð¬¤U2í- _wandbruntime‚3817  _runtime‚3817bBØVîÐ /loss/FRt1/assignment_loss‚4.138210807157599 /loss/FBt3/detection_loss‚3.8147366464138033 epoch‚3 0loss/FRt2/assignment_loss‚2.7163139604172617 1loss/SRqqt1/assignment_loss‚2.129901686890258 1loss/SRqqt3/assignment_loss‚2.049225219795781 /loss/FRt1/detection_loss‚2.4798948823109916 .loss/FRt3/detection_loss‚2.719976334108247 .loss/FRt4/detection_loss‚2.163734162847201 trainer/global_step‚4149 /loss/FRt4/assignment_loss‚3.039591258647395 0loss/FBt3/assignment_loss‚0.4718912422657013 /loss/FRt2/detection_loss‚1.7495252478094596 /loss/FRt3/assignment_loss‚4.632580459117889 %loss/total_loss‚49.65993100937153 0loss/SRqqt2/detection_loss‚1.629075629711151 /loss/FBt2/detection_loss‚0.7591167643131485 $ _timestamp‚1.788444100469936e+09 2loss/SRqqt2/assignment_loss‚1.3512425670772792 2loss/SRqqt4/assignment_loss‚1.8714096161956881 0loss/FBt1/assignment_loss‚0.3833121200977158 1loss/FBt4/assignment_loss‚0.37234419477837427 _step‚340 1loss/FBt2/assignment_loss‚0.18204020941391263 /loss/FBt1/detection_loss‚2.3473609306595544 /loss/FBt4/detection_loss‚2.6239416003227234 1loss/SRqqt1/detection_loss‚2.2333926128016577 0loss/SRqqt3/detection_loss‚2.559573317727735 0loss/SRqqt4/detection_loss‚1.241539538491006 _runtime‚3817.517058221ÿ¥4[ïÕ /loss/FRt1/detection_loss‚2.4798948823109916 .loss/FRt3/detection_loss‚2.719976334108247 .loss/FRt4/detection_loss‚2.163734162847201 trainer/global_step‚4149 /loss/FRt4/assignment_loss‚3.039591258647395 0loss/FBt3/assignment_loss‚0.4718912422657013 /loss/FRt2/detection_loss‚1.7495252478094596 /loss/FRt3/assignment_loss‚4.632580459117889 %loss/total_loss‚49.65993100937153 0loss/SRqqt2/detection_loss‚1.629075629711151 /loss/FBt2/detection_loss‚0.7591167643131485 $ _timestamp‚1.788444100469936e+09 2loss/SRqqt2/assignment_loss‚1.3512425670772792 2loss/SRqqt4/assignment_loss‚1.8714096161956881 0loss/FBt1/assignment_loss‚0.3833121200977158 1loss/FBt4/assignment_loss‚0.37234419477837427 _step‚340 1loss/FBt2/assignment_loss‚0.18204020941391263 /loss/FBt1/detection_loss‚2.3473609306595544 /loss/FBt4/detection_loss‚2.6239416003227234 1loss/SRqqt1/detection_loss‚2.2333926128016577 0loss/SRqqt3/detection_loss‚2.559573317727735 0loss/SRqqt4/detection_loss‚1.241539538491006 _runtime‚3817.517058221 /loss/FRt1/assignment_loss‚4.138210807157599 /loss/FBt3/detection_loss‚3.8147366464138033 epoch‚3 0loss/FRt2/assignment_loss‚2.7163139604172617 1loss/SRqqt1/assignment_loss‚2.129901686890258 1loss/SRqqt3/assignment_loss‚2.049225219795781Ô)Zï“ð‚(:ˆ ËûåÔßú¤Ç gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.274' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04]]ñ‚(:Ò ËûåÔî‹·É* proc.memory.percent‚16.637396812438965$ disk./.usageGB‚295.1743698120117 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125# disk.dm-0.out‚16222.66259765625 disk.sda.out‚ 142.32421875# proc.memory.rssMB‚ 2725.87109375 network.recv‚ 1.410584e+06* disk./.usagePercent‚19.809399625876733 disk.rbd1.out‚ 27.0302734375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚16222.66259765625 disk.rbd0.out‚0$ memory_percent‚95.78850269317627 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚16222.66259765625 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 cpu‚25.280705541460193 network.sent‚ 9.986067e+06 disk.dm-1.out‚ 142.32421875 disk.drbd2266.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 690.01171875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 142.32421875ô5î\“ò‚(:ˆ ÚûåÔµî§Ç gpu.0.gpu‚31.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚191.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.2176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“âÿ [ó‚(:Ð ÚûåÔîÌÉ( proc.memory.availableMB‚ 689.76171875 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2725.87109375 network.recv‚ 1.41702e+06 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 143.00390625 disk.drbd2265.out‚0 cpu‚25.27987922444542 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚16260.59619140625 network.sent‚ 1.0094723e+07) disk./.usagePercent‚19.80976469262056$ disk./.usageGB‚295.1798095703125 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 27.1552734375 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0$ memory_percent‚95.79002857208252* proc.memory.percent‚16.637396812438965 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚16260.59619140625 disk.sda3.out‚ 143.00390625# disk.dm-0.out‚16260.59619140625 disk.sda.out‚ 143.00390625_í~`“ô‚(:ˆ éûåÔ¹½ÅÇ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ_Rõ‚(:Ç éûåÔ âªÊ network.recv‚ 1.420753e+06 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 143.55078125 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125$ disk./.usageGB‚295.2038230895996 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚16319.32421875 disk.dm-1.in‚0 disk.drbd2265.out‚0$ memory_percent‚95.78852653503418# proc.memory.rssMB‚ 2725.87109375 network.sent‚ 1.0099978e+07 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.out‚ 143.55078125 disk.drbd1716.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.637396812438965 proc.cpu.threads‚17( disk./.usagePercent‚19.8113762600486 disk.sda.out‚ 143.55078125# disk.nvme0n1.out‚16319.32421875' proc.memory.availableMB‚ 690.0078125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚0.078125 disk.dm-0.out‚16319.32421875 disk.rbd1.out‚ 27.2177734375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.2789743614020538Í°®2ö- _wandbruntime‚3859  _runtime‚3859=ƒ½È÷ ' lr-AdamW/pg1‚0.00012707905204157121 ' lr-AdamW/pg2‚0.00012707905204157121 trainer/global_step‚4199 $ _timestamp‚1.788444142927395e+09 _runtime‚3859.974645064 _step‚341%£bÍøÇ ' lr-AdamW/pg1‚0.00012707905204157121 ' lr-AdamW/pg2‚0.00012707905204157121 trainer/global_step‚4199 $ _timestamp‚1.788444142927395e+09 _runtime‚3859.974645064 _step‚341ÕÝa=2ù- _wandbruntime‚3859  _runtime‚3859éÒ.µò'úìO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296800816494080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91352703 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43554114 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74054970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326599925508096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202535936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3954680014834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330777880118784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292692063723886 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292691265483762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300771484146688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165430785 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65577845 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3970650482176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3969867356672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3969867356672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91375813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165407673 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300771484146688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884441429281383e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3970650482176 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300770683850752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300770683850752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330777652810240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74054970 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296800816494080 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296800833664512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3859.975596801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43554111 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227308544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300770683850752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300770683850752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330777652810240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74054970 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296800816494080 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296800833664512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3859.975596801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43554111 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227308544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326599925508096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202535936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3954680014834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330777880118784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292692063723886 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292691265483762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚2ºDD¶ 7968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300771484146688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165430785 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65577845 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3970650482176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3969867356672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3969867356672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91375813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165407673 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300771484146688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884441429281383e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3970650482176 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165469185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43564103 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296869465238016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65592883 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296869465238016 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300841043218944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 _step‚343 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330854307864576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326675607698432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292759007229094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65592334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192036864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300840242909696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3971577980928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296869448067584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216809472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91397173 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43564106 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22028231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300841043218944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165446047 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74072012 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165469185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91397173 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3970794842112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165446047 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3971577980928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292759805469426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288803420297216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91374037 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74072012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3970794842112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3955603971238 _runtime‚3860.810117803%«ÝS'þûM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165469185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43564103 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296869465238016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65592883 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296869465238016 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300841043218944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 _step‚343 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330854307864576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326675607698432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292759007229094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65592334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192036864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300840242909696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3971577980928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296869448067584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216809472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91397173 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43564106 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22028231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300841043218944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165446047 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74072012 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165469185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91397173 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3970794842112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165446047 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3971577980928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292759805469426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288803420297216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91374037 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74072012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3970794842112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3955603971238 _runtime‚3860.810117803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4178916975616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0×E \Q2ÿ- _wandbruntime‚3860  _runtime‚3860 ¶U€Ï 0loss/FBt4/assignment_loss‚0.8704028030236562 1loss/SRqqt4/detection_loss‚1.2802170034911897 /loss/FBt2/detection_loss‚0.6015884451540618 epoch‚3 trainer/global_step‚4199 .loss/FRt1/detection_loss‚2.538084030151367 /loss/FRt1/assignment_loss‚4.123241572082042 1loss/FBt2/assignment_loss‚0.19307746209144483 0loss/SRqqt2/detection_loss‚1.567980143122184 /loss/FBt1/detection_loss‚2.8406881541013718 &loss/total_loss‚51.071512402949004 0loss/FBt3/assignment_loss‚0.7818851235645395 .loss/FBt3/detection_loss‚2.194509227360998 % _timestamp‚1.7884441437640269e+09 _step‚344 /loss/FRt3/assignment_loss‚4.782903727363138 2loss/SRqqt2/assignment_loss‚1.5930067064145055 _runtime‚3860.811183444 2loss/SRqqt1/assignment_loss‚2.1589978881504224 1loss/SRqqt3/detection_loss‚2.1665243917220347 -loss/FBt4/detection_loss‚2.52197704050276 1loss/SRqqt1/detection_loss‚2.2539511037909468 /loss/FRt4/assignment_loss‚3.400948388903749 .loss/FRt4/detection_loss‚1.994416801368489 /loss/FRt2/detection_loss‚1.9912029038718406 /loss/FRt2/assignment_loss‚3.303150033697169 1loss/SRqqt3/assignment_loss‚2.578741669654846 1loss/SRqqt4/assignment_loss‚1.839725917743312 0loss/FBt1/assignment_loss‚0.8361373911611558 /loss/FRt3/detection_loss‚2.6581544744617798›Ø×xZÔ /loss/FRt2/detection_loss‚1.9912029038718406 /loss/FRt2/assignment_loss‚3.303150033697169 1loss/SRqqt3/assignment_loss‚2.578741669654846 1loss/SRqqt4/assignment_loss‚1.839725917743312 0loss/FBt1/assignment_loss‚0.8361373911611558 /loss/FRt3/detection_loss‚2.6581544744617798 0loss/FBt4/assignment_loss‚0.8704028030236562 1loss/SRqqt4/detection_loss‚1.2802170034911897 /loss/FBt2/detection_loss‚0.6015884451540618 epoch‚3 trainer/global_step‚4199 .loss/FRt1/detection_loss‚2.538084030151367 /loss/FRt1/assignment_loss‚4.123241572082042 1loss/FBt2/assignment_loss‚0.19307746209144483 0loss/SRqqt2/detection_loss‚1.567980143122184 /loss/FBt1/detection_loss‚2.8406881541013718 &loss/total_loss‚51.071512402949004 0loss/FBt3/assignment_loss‚0.7818851235645395 .loss/FBt3/detection_loss‚2.194509227360998 % _timestamp‚1.7884441437640269e+09 _step‚344 /loss/FRt3/assignment_loss‚4.782903727363138 2loss/SRqqt2/assignment_loss‚1.5930067064145055 _runtime‚3860.811183444 2loss/SRqqt1/assignment_loss‚2.1589978881504224 1loss/SRqqt3/detection_loss‚2.1665243917220347 -loss/FBt4/detection_loss‚2.52197704050276 1loss/SRqqt1/detection_loss‚2.2539511037909468 /loss/FRt4/assignment_loss‚3.400948388903749 .loss/FRt4/detection_loss‚1.994416801368489Ø1½¿ž‚‚(:“ øûåÔÈñ¼Ç gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÀª;ƒ‚(:° øûåÔ³ÑÓÉ$ disk./.usageGB‚295.2570343017578 disk.sda3.in‚0* proc.memory.percent‚16.637396812438965 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 network.sent‚ 1.0106012e+07 network.recv‚ 1.424812e+06 disk.dm-1.out‚144! disk.dm-0.out‚16357.490234375 disk.drbd2266.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚16357.490234375 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2725.87109375 cpu‚25.27810209987623 proc.cpu.threads‚17* disk./.usagePercent‚19.814947309143722 disk.dm-0.in‚0.078125 disk.nvme0c0n1.in‚0.078125' proc.memory.availableMB‚ 689.9765625$ disk.nvme0n1.out‚16357.490234375 disk.sda3.out‚144 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 27.2177734375 disk.sda.out‚144$ memory_percent‚95.78871726989746 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0Ÿ¬X÷ž„‚(:“ ‡üåÔÊ‘²Ç gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.748799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j ÜW…‚(:Ì ‡üåÔ²É¬É disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0* disk./.usagePercent‚19.817908804299815 disk.drbd2265.in‚0 disk.dm-1.out‚ 144.96484375 disk.drbd1716.out‚0 network.recv‚ 1.430921e+06 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 144.96484375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2725.87109375 disk.sda3.out‚ 144.96484375 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.277231336935074 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚27.3125 disk.sda1.out‚0$ memory_percent‚95.78866958618164% disk./.usageGB‚295.30116271972656 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚0.078125( disk.nvme0c0n1.out‚16409.14599609375& proc.memory.availableMB‚ 689.984375& disk.nvme0n1.out‚16409.14599609375# disk.dm-0.out‚16409.14599609375* proc.memory.percent‚16.637396812438965 network.sent‚ 1.0214521e+07 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0ދ㑆‚(:† –üåÔ¸Î–Ç gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dì—R‡‚(:Ç –üåÔ—œ¦Ê disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 145.6328125 disk.dm-1.out‚ 145.6328125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0n1.out‚16456.09912109375 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.87109375 network.recv‚ 1.435293e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚19.820650132976457$ memory_percent‚95.78876495361328 disk.sda1.out‚0$ disk./.usageGB‚295.3420104980469 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.rbd1.out‚27.4375 disk.nvme0c0n1.in‚0.078125( disk.nvme0c0n1.out‚16456.09912109375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 689.96875* proc.memory.percent‚16.637396812438965 cpu‚25.276350580602656# disk.dm-0.out‚16456.09912109375 disk.drbd2266.in‚0 disk.sda.out‚ 145.6328125 network.sent‚ 1.0220775e+07 disk.drbd2080.out‚0 disk.rbd0.in‚0ìVð2ˆ- _wandbruntime‚3903  _runtime‚3903U’Á ljÁ trainer/global_step‚4249 % _timestamp‚1.7884441867733903e+09 _runtime‚3903.820704607 _step‚345 & lr-AdamW/pg1‚0.0001262288653869038 & lr-AdamW/pg2‚0.0001262288653869038 M<ãÌŠÆ trainer/global_step‚4249 % _timestamp‚1.7884441867733903e+09 _runtime‚3903.820704607 _step‚345 & lr-AdamW/pg1‚0.0001262288653869038 & lr-AdamW/pg2‚0.0001262288653869038Ù˜!º2‹- _wandbruntime‚3903  _runtime‚3903øó[ÃŒìO _runtime‚3903.821628912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334610911987200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22275720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300232412339712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300232395169280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223114240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92445865 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167352837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300232395169280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292077491954688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22275184 ZLDeviceA\íiÞStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44053711 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304249451507712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92445865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292077474915328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167329725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4017039168000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4016256042496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296078366341570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74906970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304248651211776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44053714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300232412339712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66329434 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66328895 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74906972 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330384035024896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296079164581694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304248651211776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334610688872960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304249451507712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 trainer/global_step‚4249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330384010252288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4000891426242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4017039168000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4001672627006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4016256042496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0û·ÉS÷'ñO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4000891426242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4017039168000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4001672627006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4016256042496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3903.821628912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334610911987200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22275720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300232412339712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300232395169280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223114240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92445865 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167352837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300232395169280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292077491954688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22275184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44053711 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304249451507712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92445865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292077474915328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167329725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4017039168000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4016256042496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296078366341570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74906970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304248651211776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44053714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300232412339712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66329434 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66328895 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74906972 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330384035024896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296079164581694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304248651211776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334610688872960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304249451507712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 trainer/global_step‚4249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330384010252288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 Jú&ôM B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300301043913216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22280118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330459717215232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4017966666752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92444089 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227663838208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296146108086902 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296146906327234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚347 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44063706 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22280644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330459692442624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292144292704256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183648256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4227847486464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66343821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 _runtime‚3904.665252639 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304319010579968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292144309743616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4001815382646 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167391237 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4017966666752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334687356280832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44063703 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74924010 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300301026742784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167391237 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74924012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92467225 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4017183527936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 trainer/global_step‚4249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92444089 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4002596583618 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300301026742784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304318210270720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4017183527936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 # _timestamp‚1.78844418761728e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167368099 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74924010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608‹p’`ÜùM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4227847486464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66343821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 _runtime‚3904.665252639 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304319010579968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292144309743616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4001815382646 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167391237 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4017966666752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334687356280832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44063703 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74924010 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300301026742784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167391237 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74924012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92467225 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4017183527936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 trainer/global_step‚4249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92444089 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4002596583618 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300301026742784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304318210270720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4017183527936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 # _timestamp‚1.78844418761728e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167368099 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74924010 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300301043913216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22280118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330459717215232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4017966666752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92444089 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227663838208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296146108086902 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296146906327234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚347 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44063706 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22280644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330459692442624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292144292704256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183648256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489Û¬û492‘- _wandbruntime‚3904  _runtime‚39044"½åV’Ð 1loss/SRqqt2/detection_loss‚1.7340783833554296 trainer/global_step‚4249 _runtime‚3904.666411396 1loss/SRqqt3/detection_loss‚2.8172127421085653 epoch‚3 0loss/FBt3/assignment_loss‚0.3185307462998545 /loss/FRt3/detection_loss‚2.6564218574448635 0loss/SRqqt1/detection_loss‚1.996726960728043 1loss/SRqqt4/detection_loss‚1.3643166235027213 /loss/FRt1/assignment_loss‚4.069073594751812 2loss/SRqqt1/assignment_loss‚2.0177861726597737 1loss/SRqqt2/assignment_loss‚1.758562782497117 1loss/SRqqt3/assignment_loss‚2.228115235383694 2loss/SRqqt4/assignment_loss‚1.8933583966766794 0loss/FBt4/assignment_loss‚0.4820739544928074 /loss/FRt4/detection_loss‚2.2067217575876334 /loss/FRt2/assignment_loss‚3.483811864386434 0loss/FBt1/assignment_loss‚0.5773540884256363 -loss/FRt3/assignment_loss‚4.9932266505141 .loss/FBt1/detection_loss‚2.450367547571659 % _timestamp‚1.7884441876192563e+09 0loss/FBt2/assignment_loss‚0.2899311841094985 /loss/FBt2/detection_loss‚0.6386408339651892 /loss/FBt3/detection_loss‚2.3992572458727017 &loss/total_loss‚51.535455128623035 /loss/FBt4/detection_loss‚2.6938006460666655 0loss/FRt4/assignment_loss‚3.9077017736016657 _step‚348 .loss/FRt1/detection_loss‚2.663734262897855 /loss/FRt2/detection_loss‚1.8946498237226321à’7Ï[“Õ 0loss/FBt2/assignment_loss‚0.2899311841094985 /loss/FBt2/detection_loss‚0.6386408339651892 /loss/FBt3/detection_loss‚2.3992572458727017 &loss/total_loss‚51.535455128623035 /loss/FBt4/detection_loss‚2.6938006460666655 0loss/FRt4/assignment_loss‚3.9077017736016657 _step‚348 .loss/FRt1/detection_loss‚2.663734262897855 /loss/FRt2/detection_loss‚1.8946498237226321 1loss/SRqqt2/detection_loss‚1.7340783833554296 trainer/global_step‚4249 _runtime‚3904.666411396 1loss/SRqqt3/detection_loss‚2.8172127421085653 epoch‚3 0loss/FBt3/assignment_loss‚0.3185307462998545 /loss/FRt3/detection_loss‚2.6564218574448635 0loss/SRqqt1/detection_loss‚1.996726960728043 1loss/SRqqt4/detection_loss‚1.3643166235027213 /loss/FRt1/assignment_loss‚4.069073594751812 2loss/SRqqt1/assignment_loss‚2.0177861726597737 1loss/SRqqt2/assignment_loss‚1.758562782497117 1loss/SRqqt3/assignment_loss‚2.228115235383694 2loss/SRqqt4/assignment_loss‚1.8933583966766794 0loss/FBt4/assignment_loss‚0.4820739544928074 /loss/FRt4/detection_loss‚2.2067217575876334 /loss/FRt2/assignment_loss‚3.483811864386434 0loss/FBt1/assignment_loss‚0.5773540884256363 -loss/FRt3/assignment_loss‚4.9932266505141 .loss/FBt1/detection_loss‚2.450367547571659 % _timestamp‚1.7884441876192563e+09ÜõÇd“”‚(:ˆ ¥üåÔ¸…ÇÇ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ˆFL•‚(:Á ¥üåÔüÑÜÉ disk.dm-1.in‚0 disk.drbd1716.out‚0 network.recv‚ 1.439045e+06 disk.drbd2265.in‚0 cpu‚25.275496552637602 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0% proc.memory.availableMB‚ 689.90625* proc.memory.percent‚16.637396812438965 proc.cpu.threads‚17 disk.sda2.in‚0$ disk./.usageGB‚295.4049644470215 disk.drbd2080.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2725.87109375 disk.drbd2266.in‚0 disk.dm-1.out‚ 146.24609375$ disk.nvme0n1.out‚16503.794921875 disk.nvme0c0n1.in‚0.078125! disk.dm-0.out‚16503.794921875* disk./.usagePercent‚19.824875025314025 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 146.24609375 disk.dm-0.in‚0.078125 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚27.5 disk.rbd0.out‚0& disk.nvme0c0n1.out‚16503.794921875 disk.nvme0n1.in‚0.078125$ memory_percent‚95.78914642333984 network.sent‚ 1.0226077e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 146.24609375œV“–‚(:ˆ ´üåÔçûÑÇ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ/ÆM—‚(: ´üåÔÍñ¹É disk.rbd1.out‚27.5 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125# memory_percent‚95.7876443862915 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 network.recv‚ 1.445288e+06* disk./.usagePercent‚19.826606404268095$ disk./.usageGB‚295.4307632446289$ disk.nvme0n1.out‚16543.896484375 disk.dm-0.in‚0.078125 disk.drbd2080.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0334474e+07& disk.nvme0c0n1.out‚16543.896484375 disk.sda.out‚ 146.91796875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 146.91796875 disk.dm-1.in‚0 disk.dm-1.out‚ 146.91796875! disk.dm-0.out‚16543.896484375 disk.drbd2266.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 690.15234375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 cpu‚25.27464516392612 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.87109375* proc.memory.percent‚16.637396812438965 disk.drbd2080.out‚0—Eý0“˜‚(:ˆ ÃüåÔî¯ÑÇ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.1442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ó_£L™‚(:Á ÃüåÔ£¹ëÉ disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.449958e+06 disk.sda.out‚ 147.7421875# proc.memory.rssMB‚ 2725.87109375 disk.dm-1.in‚0 network.sent‚ 1.0340657e+07) disk./.usagePercent‚19.83131079871729 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0% disk./.usageGB‚295.50086212158203 disk.dm-0.out‚ 16596.453125 disk.sda1.in‚0! disk.nvme0n1.out‚ 16596.453125( proc.memory.availableMB‚ 689.66015625* proc.memory.percent‚16.637396812438965 cpu‚25.273791475852047 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 147.7421875 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 27.5322265625# disk.nvme0c0n1.out‚ 16596.453125 disk.dm-1.out‚ 147.7421875 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.sda1.out‚0$ memory_percent‚95.79064846038818 <’2š- _wandbruntime‚3947  _runtime‚39478Ö¤­È›Â _step‚349 ' lr-AdamW/pg1‚0.00012536842369172054 ' lr-AdamW/pg2‚0.00012536842369172054 trainer/global_step‚4299 $ _timestamp‚1.788444230691176e+09 _runtime‚3947.738416771# À/ÍœÇ $ _timestamp‚1.788444230691176e+09 _runtime‚3947.738416771 _step‚349 ' lr-AdamW/pg1‚0.00012536842369172054 ' lr-AdamW/pg2‚0.00012536842369172054 trainer/global_step‚4299Ýòýˆ2- _wandbruntime‚3947  _runtime‚3947«Ûvjò'žìO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93492807 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307726618572800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44553314 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334168119769088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4063427853824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4062644728320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75758970 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4063427853824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216822784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169251777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303663973844480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22526094 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4275885753856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67079408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚307726618572800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚307727418868736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4275693703680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299465467199378 trainer/global_step‚4299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295418381401088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334168144541696 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22525565 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884442306920216e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295418364361728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4062644728320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299466265439502 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4047102837650 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93515917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93492807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303663973844480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75758972 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93515917 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75758972 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75758970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44553311 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4047884038414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334168144541696 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22525565 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884442306920216e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295418364361728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4062644728320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299466265439502 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4047102837650 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93515917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93492807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303663973844480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75758972 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93515917 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75758972 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75758970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44553311 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4047884038414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307726618572800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44553314 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 aMDevi®ç8ëöceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334168119769088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4063427853824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4062644728320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75758970 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4063427853824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216822784ÞÈh2 - _wandbruntime‚3948  _runtime‚3948Tî‚Ÿü&¡öM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67093748 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4064355352576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22530445 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4276669937664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307796977940992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307796177631744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93537277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44563303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22530975 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4276859877376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295485199190016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3948.571456307 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299534007185042 % _timestamp‚1.7884442315234005e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048807995026 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93514141 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307796977940992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334243826732032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75776012 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75776010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299533208944710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75776010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4063572213760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338520471897088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048026794054 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93514141 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4063572213760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 _step‚351 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169290151 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313289 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93537277 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303732622588416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4064355352576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169290151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75776012 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67093748 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4064355352576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22530445 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4276669937664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307796977940992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307796177631744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93537277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44563303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22530975 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4276859877376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295485199190016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3948.571456307 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299534007185042 % _timestamp‚1.7884442315234005e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048807995026 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93514141 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307796977940992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334243826732032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75776012 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75776010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299533208944710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75776010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4063572213760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338520471897088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048026794054 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93514141 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4063572213760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 _step‚351 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169290151 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313289 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93537277 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303732622588416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4064355352576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169290151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75776012 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94562859 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4324812336128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4109033414144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298759270847488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94585969 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307095569690112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307095569690112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4094095449822 trainer/global_step‚4349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311205386229760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22777360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3lIô2%71895808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76610970 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307095552519680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311205386229760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76610972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4109033414144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 $ _timestamp‚1.788444275036207e+09 _runtime‚3992.083711177 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94585969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337952229285888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67830271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45052914 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337952254058496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298759253808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76610970 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311204585933824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22777887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76610972 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94562859 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45052911 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67830801 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171173829 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307095552519680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4324622383104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109816539648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311204585933824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302852568057186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093314249058 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302853366297310 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328%9‚—ö'±ðO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4094095449822 trainer/global_step‚4349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311205386229760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22777360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76610970 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307095552519680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311205386229760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76610972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4109033414144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 $ _timestamp‚1.788444275036207e+09 _runtime‚3992.083711177 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94585969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337952229285888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67830271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45052914 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337952254058496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298759253808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76610970 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311204585933824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22777887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76610972 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94562859 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45052911 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67830801 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171173829 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307095552519680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189953024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4324622383104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109816539648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311204585933824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302852568057186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093314249058 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302853366297310 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171196941 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342277066394624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94562859 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4324812336128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4109033414144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298759270847488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94585969 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307095569690112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214725632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307095569690112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264â"; Ÿ2²- _wandbruntime‚3992  _runtime‚3992ë¥ñû&³õM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94584193 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67845780 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚355 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4325582156288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 3992.90604661 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338027911476224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302920309802518 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298826071597056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338027936248832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 % _timestamp‚1.7884442758582647e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298826088636416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76628010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110744038400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22782874 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171235341 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94607329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45062906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311274945302016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342353718830592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225198080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4325782581760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4095019406434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67845246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109960899584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592b»¼`•´úM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94584193 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67845780 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚355 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4325582156288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 3992.90604661 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338027911476224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302920309802518 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298826071597056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338027936248832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 % _timestamp‚1.7884442758582647e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298826088636416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76628010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110744038400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22782874 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171235341 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94607329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45062906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311274945302016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342353718830592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225198080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4325782581760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4095019406434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67845246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109960899584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332ãÙúc…2µ- _wandbruntime‚3992  _runtime‚3992}¬ïUZ¶Ô % _timestamp‚1.7884442758602703e+09 /loss/FRt1/assignment_loss‚4.152009675377293 .loss/FRt3/assignment_loss‚4.77968032019479 /loss/FRt4/assignment_loss‚3.154850172996521 2loss/SRqqt3/assignment_loss‚2.2172520745545627 /loss/FBt3/detection_loss‚2.4785126989538018 _runtime‚3992.907432146 0loss/FBt3/assignment_loss‚0.4587492617166307 /loss/FRt2/detection_loss‚1.8975140661001206 1loss/SRqqt3/detection_loss‚2.3163030952215196 1loss/SRqqt4/detection_loss‚1.2700528866052627 2loss/SRqqt1/assignment_loss‚1.9557118219764609 0loss/FBt1/assignment_loss‚0.4916693406800429 1loss/SRqqt2/detection_loss‚1.7884994968771935 /loss/FBt2/detection_loss‚0.7205442086654417 2loss/SRqqt4/assignment_loss‚1.9298511818051338 /loss/FRt1/detection_loss‚2.5495585712947344 epoch‚3 trainer/global_step‚4349 0loss/FBt4/assignment_loss‚0.7600519231387547 /loss/FRt3/detection_loss‚2.7994617121560235 /loss/FBt1/detection_loss‚3.5190191666285195 /loss/FBt4/detection_loss‚2.8893942747797285 0loss/FRt2/assignment_loss‚3.2566652461886405 1loss/SRqqt2/assignment_loss‚2.245045557618141 0loss/FBt2/assignment_loss‚0.1964273639467125 .loss/FRt4/detection_loss‚2.171902222186327 1loss/SRqqt1/detection_loss‚2.2611842900514603 &loss/total_loss‚52.259910629713815 _step‚356¨£+£_·Ù /loss/FBt3/detection_loss‚2.4785126989538018 _runtime‚3992.907432146 0loss/FBt3/assignment_loss‚0.4587492617166307 /loss/FRt2/detection_loss‚1.8975140661001206 1loss/SRqqt3/detection_loss‚2.3163030952215196 1loss/SRqqt4/detection_loss‚1.2700528866052627 2loss/SRqqt1/assignment_loss‚1.9557118219764609 0loss/FBt1/assignment_loss‚0.4916693406800429 1loss/SRqqt2/detection_loss‚1.7884994968771935 /loss/FBt2/detection_loss‚0.7205442086654417 2loss/SRqqt4/assignment_loss‚1.9298511818051338 /loss/FRt1/detection_loss‚2.5495585712947344 epoch‚3 trainer/global_step‚4349 0loss/FBt4/assignment_loss‚0.7600519231387547 /loss/FRt3/detection_loss‚2.7994617121560235 /loss/FBt1/detection_loss‚3.5190191666285195 /loss/FBt4/detection_loss‚2.8893942747797285 0loss/FRt2/assignment_loss‚3.2566652461886405 1loss/SRqqt2/assignment_loss‚2.245045557618141 0loss/FBt2/assignment_loss‚0.1964273639467125 .loss/FRt4/detection_loss‚2.171902222186327 1loss/SRqqt1/detection_loss‚2.2611842900514603 &loss/total_loss‚52.259910629713815 _step‚356 % _timestamp‚1.7884442758602703e+09 /loss/FRt1/assignment_loss‚4.152009675377293 .loss/FRt3/assignment_loss‚4.77968032019479 /loss/FRt4/assignment_loss‚3.154850172996521 2loss/SRqqt3/assignment_loss‚2.2172520745545627äL¡/擸‚(:ˆ ÿüåÔ–œÑÇ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïZ˜sT¹‚(:É ÿüåÔÚËÜÉ* disk./.usagePercent‚19.841802243477993 disk.dm-1.out‚ 150.48046875 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚16794.2001953125 disk.drbd2265.in‚0& proc.memory.availableMB‚ 689.484375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 150.48046875% disk.nvme0n1.out‚16794.2001953125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda.out‚ 150.48046875 disk.sda2.out‚0$ memory_percent‚95.79172134399414 proc.memory.rssMB‚2725.875 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 27.8681640625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125$ disk./.usageGB‚295.6571922302246 network.sent‚ 1.0465709e+07 network.recv‚ 1.468167e+06 disk.dm-0.in‚0.078125' disk.nvme0c0n1.out‚16794.2001953125* proc.memory.percent‚16.637420654296875 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 cpu‚25.270399972904215Ÿ„&À“º‚(:ˆ ŽýåÔÎøˆÇ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ÑÍT»‚(:É ŽýåÔö¡§Ê disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda.out‚ 150.984375 disk.sda3.in‚0 disk.dm-1.out‚ 150.984375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 network.recv‚ 1.474471e+06# disk.dm-0.out‚16845.53076171875 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0& disk.nvme0n1.out‚16845.53076171875 disk.rbd1.out‚ 28.2587890625$ memory_percent‚95.79179286956787( proc.memory.availableMB‚ 689.47265625* proc.memory.percent‚16.637420654296875 network.sent‚ 1.0574328e+07 disk.drbd2266.in‚0 proc.memory.rssMB‚2725.875 proc.cpu.threads‚17% disk./.usageGB‚295.65618896484375( disk.nvme0c0n1.out‚16845.53076171875 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0.078125 disk.sda1.out‚0 disk.sda3.out‚ 150.984375 cpu‚25.269575445235898* disk./.usagePercent‚19.841734913496488 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125³hz“¼‚(:ˆ ýåÔÃç¤È gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûJ²T½‚(:É ýåÔì“ÁÉ% disk.nvme0n1.out‚16909.6162109375 disk.sda1.out‚0 disk.rbd0.out‚0 proc.memory.rssMB‚2725.875* proc.memory.percent‚16.637420654296875" disk.dm-0.out‚16909.6162109375 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 689.4765625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.recv‚ 1.478321e+06 disk.nvme0n1.in‚0.078125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 151.97265625 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 28.5869140625 disk.nvme0c0n1.in‚0.078125 cpu‚25.268774647650527 disk.dm-0.in‚0.078125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 151.97265625 proc.cpu.threads‚17$ disk./.usageGB‚295.7026901245117' disk.nvme0c0n1.out‚16909.6162109375 network.sent‚ 1.0579615e+07) disk./.usagePercent‚19.84485564533888 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 151.97265625$ memory_percent‚95.79176902770996Zê0`2¾- _wandbruntime‚4035  _runtime‚4035XÌÈ¿Â ' lr-AdamW/pg1‚0.00012361749453861087 ' lr-AdamW/pg2‚0.00012361749453861087 trainer/global_step‚4399 % _timestamp‚1.7884443189144487e+09 _runtime‚ 4035.96169769 _step‚357Œ¶6¥ÍÀÇ ' lr-AdamW/pg2‚0.00012361749453861087 trainer/global_step‚4399 % _timestamp‚1.7884443189144487e+09 _runtime‚ 4035.96169769 _step‚357 ' lr-AdamW/pg1‚0.00012361749453861087åN_`¦2Á- _wandbruntime‚4035  _runtime‚4035Ðéñ'ÂëO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4139525660466 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚4035.962774614 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310527148365312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45552511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346109943683584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341736338802688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302100160293888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95632911 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95632911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310527131194880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173095881 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314683353590784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23028383 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346109731055104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310527131194880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚358 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173118993 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95656021 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23027867 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4140306861230 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302100143254528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314682553294848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4156205225472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68580897 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4155422099968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4156205225472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306239668914994 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310527148365312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4155422099968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4373392252416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306240467155118 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314683353590784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 $ _timestamp‚1.788444318915234e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77462972 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4373580108288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95656021 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68580378 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328jê’ö'ÃðO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4155422099968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4156205225472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212628480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306239668914994 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310527148365312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4155422099968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4373392252416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306240467155118 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314683353590784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 $ _timestamp‚1.788444318915234e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77462972 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4373580108288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95656021 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68580378 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4139525660466 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚4035.962774614 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310527148365312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45552511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346109943683584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341736338802688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302100160293888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95632911 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95632911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310527131194880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173095881 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314683353590784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23028383 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346109731055104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310527131194880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚358 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173118993 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187855872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95656021 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23027867 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4140306861230 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302100143254528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314682553294848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4156205225472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68580897æ¢ÚkO2Ä- _wandbruntime‚4036  _runtime‚4036@Õ”«eÅöM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77480012 =7DeviceSä÷óy—&tatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310595779938816 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4156349585408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95677381 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4156349585408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45562506 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173157393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314752912663040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306308208900658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314752112353792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302166961043456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341812045765632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310595779938816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173157393 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310595762768384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341812020993024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310595762768384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173134255 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68595310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521È•&›'ÆûM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314752912663040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95654245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚359 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 % _timestamp‚1.7884443197561028e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95677381 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23032807 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346186386217984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4141230817842 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4156349585408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95677381 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4156349585408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45562506 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173157393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314752912663040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306308208900658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314752112353792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302166961043456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341812045765632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310595779938816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173157393 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310595762768384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341812020993024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310595762768384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173134255 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68595310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77480012 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310595779938816 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318160520655872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4202593911296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23279681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313958709870080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349942820232192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204239872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78314972 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318161320951808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313958727040512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309627568012926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78314970 TGDeviceStatsMonitor.on_train_batch_start/inactiÚŒŒå…ve_split.small_pool.freed‚23279158 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78314972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69331795 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318160520655872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46052111 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚4080.363049341 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345520473092096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4202593911296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313958709870080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4201810785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚4080.363049341 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345520473092096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4202593911296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313958709870080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4201810785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318160520655872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4202593911296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23279681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313958709870080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349942820232192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204239872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78314972 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318161320951808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313958727040512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309627568012926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78314970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23279158 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78314972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69331795 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318160520655872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46052111 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504êÖ?-Ð2Ö- _wandbruntime‚4081  _runtime‚4081擽nü&×öM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314027358614016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96724297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46062103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23284162 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305507850489856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96724297 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314027358614016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69346809 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78332010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318230880024064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175079445 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4203521410048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4202738271232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96747433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175056307 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314027341443584 % _timestamp‚1.7884443641479313e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23284703 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4202738271232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4187442229250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4186661028278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314027341443584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175079445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78332012 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96747433 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175056307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350019259066880 _runtime‚4081.195666942 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69346265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350019482167808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318230079714816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305507867529216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78332012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318230880024064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4203521410048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78332010½ì£K3ØûM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345596130509824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175056307 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314027341443584 % _timestamp‚1.7884443641479313e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23284703 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4202738271232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4187442229250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4186661028278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314027341443584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175079445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78332012 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96747433 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175056307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350019259066880 _runtime‚4081.195666942 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69346265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350019482167808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318230079714816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305507867529216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78332012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318230880024064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4203521410048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78332010 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314027358614016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96724297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46062103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23284162 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305507850489856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96724297 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314027358614016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69346809 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78332010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318230880024064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175079445 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4203521410048 C9DeviceStatsMonitor.on_trainÅO}¿Î _batch_end/allocation.all.peak‚28217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4202738271232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96747433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 BÛZÚÔ .loss/FRt2/detection_loss‚1.943554147443873 0loss/FRt1/assignment_loss‚3.9206576725331748 1loss/SRqqt1/assignment_loss‚2.013173216746913 /loss/FRt3/assignment_loss‚4.157128425205455 1loss/FBt1/assignment_loss‚0.18419425748288631 /loss/FRt4/detection_loss‚2.6386203403499993 1loss/SRqqt2/detection_loss‚1.5735341501541626 /loss/FBt2/detection_loss‚0.6324877168147732 1loss/SRqqt4/detection_loss‚1.4613380698781264 .loss/FBt1/detection_loss‚2.407305383682251 % _timestamp‚1.7884443641495755e+09 0loss/FRt2/assignment_loss‚2.6322825665169574 0loss/FBt3/assignment_loss‚0.7067095977423681 1loss/SRqqt1/detection_loss‚2.1334736372033754 1loss/SRqqt3/detection_loss‚2.3873105798448835 /loss/FBt3/detection_loss‚2.3987398823102315 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚2.4454454345362526 0loss/FRt4/assignment_loss‚3.5634251927787606 2loss/SRqqt2/assignment_loss‚1.4530937835001028 0loss/FBt4/assignment_loss‚0.4944116519764066 -loss/FRt1/detection_loss‚2.51864905400974 _runtime‚4081.196723176 2loss/SRqqt4/assignment_loss‚2.0228168925172403 1loss/FBt2/assignment_loss‚0.17726796291411226 .loss/FRt3/detection_loss‚2.837209437699879 /loss/FBt4/detection_loss‚2.6449321866035462 %loss/total_loss‚49.34776124044547 epoch‚3 _step‚364cÚü_ÛÙ 1loss/SRqqt3/detection_loss‚2.3873105798448835 /loss/FBt3/detection_loss‚2.3987398823102315 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚2.4454454345362526 0loss/FRt4/assignment_loss‚3.5634251927787606 2loss/SRqqt2/assignment_loss‚1.4530937835001028 0loss/FBt4/assignment_loss‚0.4944116519764066 -loss/FRt1/detection_loss‚2.51864905400974 _runtime‚4081.196723176 2loss/SRqqt4/assignment_loss‚2.0228168925172403 1loss/FBt2/assignment_loss‚0.17726796291411226 .loss/FRt3/detection_loss‚2.837209437699879 /loss/FBt4/detection_loss‚2.6449321866035462 %loss/total_loss‚49.34776124044547 epoch‚3 _step‚364 .loss/FRt2/detection_loss‚1.943554147443873 0loss/FRt1/assignment_loss‚3.9206576725331748 1loss/SRqqt1/assignment_loss‚2.013173216746913 /loss/FRt3/assignment_loss‚4.157128425205455 1loss/FBt1/assignment_loss‚0.18419425748288631 /loss/FRt4/detection_loss‚2.6386203403499993 1loss/SRqqt2/detection_loss‚1.5735341501541626 /loss/FBt2/detection_loss‚0.6324877168147732 1loss/SRqqt4/detection_loss‚1.4613380698781264 .loss/FBt1/detection_loss‚2.407305383682251 % _timestamp‚1.7884443641495755e+09 0loss/FRt2/assignment_loss‚2.6322825665169574 0loss/FBt3/assignment_loss‚0.7067095977423681 1loss/SRqqt1/detection_loss‚2.1334736372033754ìûsóžÜ‚(:“ ÙýåÔ×øéÈ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¼ AÝ‚(:¶ ÙýåÔƉ¯Ê$ disk./.usageGB‚295.7647171020508 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda.out‚ 154.73828125 disk.sda3.out‚ 154.73828125 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.nvme0n1.out‚ 17097.46875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.078125# proc.memory.availableMB‚689.625 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0$ memory_percent‚95.79086303710938 cpu‚25.26557841155045" disk.nvme0c0n1.out‚ 17097.46875 disk.dm-1.out‚ 154.73828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2266.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 network.sent‚ 1.0705329e+07 disk.dm-0.out‚ 17097.46875 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.637420654296875 disk.dm-0.in‚0.078125 proc.memory.rssMB‚2725.875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 28.9228515625 network.recv‚ 1.497304e+06* disk./.usagePercent‚19.849018327845624l´,žÞ‚(:“ èýåÔæƒÄÇ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0о?â?ß‚(:´ èýåÔ©Ã£É disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.out‚17140.75 disk.sda.out‚ 155.11328125 disk.dm-0.in‚0.078125* proc.memory.percent‚16.637420654296875 disk.rbd1.out‚ 29.0087890625 disk.nvme0c0n1.in‚0.078125 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2725.875 disk.drbd2266.in‚0$ memory_percent‚95.79203128814697 network.recv‚ 1.503113e+06% disk./.usageGB‚295.79953384399414 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 155.11328125 disk.drbd1716.out‚0 disk.nvme0n1.out‚17140.75 network.sent‚ 1.0813708e+07 disk.drbd2080.out‚0 cpu‚25.264810543714095 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 155.11328125 disk.drbd1716.in‚0( proc.memory.availableMB‚ 689.43359375* disk./.usagePercent‚19.851354908610634 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.out‚17140.75y}A2à- _wandbruntime‚4124  _runtime‚4124ûηMÉáà _step‚365 ' lr-AdamW/pg1‚0.00012182772407024416 ' lr-AdamW/pg2‚0.00012182772407024416 trainer/global_step‚4499 % _timestamp‚1.7884444076814253e+09 _runtime‚4124.728706101e¥Þ&ÎâÈ ' lr-AdamW/pg1‚0.00012182772407024416 ' lr-AdamW/pg2‚0.00012182772407024416 trainer/global_step‚4499 % _timestamp‚1.7884444076814253e+09 _runtime‚4124.728706101 _step‚365íjä 2ã- _wandbruntime‚4124  _runtime‚4124¢`´Õò'äìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321639288312832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317390305715712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23529910 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 % _timestamp‚1.7884444076821826e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317390288545280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4248982597120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321638488016896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79166970 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46551711 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349304557836288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4471205352448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308781922147328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308781939186688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313014668870734 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4248199471616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46551714 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79166972 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79166970 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4248982597120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4471009107968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321638488016896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4232729684046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4231948483282 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79166972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317390288545280 _step‚366 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353775566944256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321639288312832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176939985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97773015 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592N‡Óž÷'åñO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79166972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317390288545280 _step‚366 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353775566944256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321639288312832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176939985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97773015 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321639288312832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317390305715712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23529910 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 % _timestamp‚1.7884444076821826e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317390288545280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4248982597120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321638488016896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79166970 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46551711 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349304557836288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4471205352448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308781922147328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308781939186688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313014668870734 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4248199471616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46551714 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79166972 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79166970 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4248982597120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4471009107968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321638488016896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4232729684046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4231948483282 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0î¹iÁùžæ‚(:“ ÷ýåÔ·‘¶Ç gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.367399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fLç‚(:Á ÷ýåÔµëÌÉ* disk./.usagePercent‚19.853404504929763# disk.dm-0.out‚17192.52490234375 disk.drbd2266.in‚0 disk.sda1.out‚0 network.sent‚ 1.0819489e+07 network.recv‚ 1.507201e+06 disk.rbd1.out‚ 29.0087890625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚17192.52490234375 disk.sda.out‚155.9375 disk.sda3.out‚155.9375 disk.drbd1716.in‚0$ memory_percent‚95.79205513000488% disk./.usageGB‚295.83007431030273 disk.dm-0.in‚0.078125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125' proc.memory.availableMB‚ 689.4296875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚155.9375 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚17192.52490234375 proc.memory.rssMB‚2725.875 disk.drbd2080.out‚0 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.637420654296875 cpu‚25.26403019059353 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125Ðúò2è- _wandbruntime‚4125  _runtime‚4125÷¦~ éöM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79184012 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353852439823872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4249910095872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177001497 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317458920118784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79184010 ?këM>ï 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4249126957056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46561703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70096040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313082410616274 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353852227208704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4232872439686 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70096586 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177001497 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97817485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97817485 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317458937289216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317458920118784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚367 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 _runtime‚4125.563262763 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321708847385088 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79184010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884444085154307e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321708047075840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4472175024640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4471987182080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23534337 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79184012 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4249910095872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46561706 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79184010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884444085154307e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321708047075840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4472175024640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4471987182080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23534337 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79184012 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4249910095872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46561706 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177001497 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317458920118784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79184010 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4249126957056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46561703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70096040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313082410616274 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353852227208704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4232872439686 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70096586 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177001497 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97817485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97817485 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317458937289216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317458920118784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 _step‚367ï¤hc¸2ë- _wandbruntime‚4125  _runtime‚4125‘`m€VìÐ .loss/FBt1/detection_loss‚5.983973532915115 trainer/global_step‚4499 1loss/SRqqt3/assignment_loss‚2.309823333285749 _step‚368 _runtime‚ 4125.56476042 0loss/FRt1/assignment_loss‚3.7149651313529297 .loss/FRt2/assignment_loss‚3.04024924437205 0loss/FBt3/assignment_loss‚0.8923820046087106 % _timestamp‚1.7884444085176167e+09 /loss/FRt4/assignment_loss‚2.741017785209876 2loss/SRqqt1/assignment_loss‚2.1142057786385218 2loss/SRqqt4/assignment_loss‚2.0323354048797717 /loss/FRt2/detection_loss‚2.0136144237385856 /loss/FRt4/detection_loss‚2.1930551076164613 /loss/FBt2/detection_loss‚0.7135458145211526 /loss/FRt1/detection_loss‚2.6949672576259163 1loss/FBt1/assignment_loss‚0.05263830107287504 0loss/FBt2/assignment_loss‚0.2108372761802882 0loss/FBt4/assignment_loss‚0.5110801290720657 0loss/SRqqt1/detection_loss‚2.478089225292206 1loss/SRqqt2/detection_loss‚1.4672861192907607 epoch‚3 0loss/FRt3/assignment_loss‚3.9801942884922026 0loss/SRqqt3/detection_loss‚2.530753867700696 0loss/SRqqt4/detection_loss‚1.225204893352034 .loss/FBt3/detection_loss‚2.124060014883677 %loss/total_loss‚53.97407307509193 2loss/SRqqt2/assignment_loss‚1.6731228744345052 /loss/FRt3/detection_loss‚3.4542999625205995 /loss/FBt4/detection_loss‚3.8223713040351868Ò¶ù“[íÕ 0loss/SRqqt3/detection_loss‚2.530753867700696 0loss/SRqqt4/detection_loss‚1.225204893352034 .loss/FBt3/detection_loss‚2.124060014883677 %loss/total_loss‚53.97407307509193 2loss/SRqqt2/assignment_loss‚1.6731228744345052 /loss/FRt3/detection_loss‚3.4542999625205995 /loss/FBt4/detection_loss‚3.8223713040351868 .loss/FBt1/detection_loss‚5.983973532915115 trainer/global_step‚4499 1loss/SRqqt3/assignment_loss‚2.309823333285749 _step‚368 _runtime‚ 4125.56476042 0loss/FRt1/assignment_loss‚3.7149651313529297 .loss/FRt2/assignment_loss‚3.04024924437205 0loss/FBt3/assignment_loss‚0.8923820046087106 % _timestamp‚1.7884444085176167e+09 /loss/FRt4/assignment_loss‚2.741017785209876 2loss/SRqqt1/assignment_loss‚2.1142057786385218 2loss/SRqqt4/assignment_loss‚2.0323354048797717 /loss/FRt2/detection_loss‚2.0136144237385856 /loss/FRt4/detection_loss‚2.1930551076164613 /loss/FBt2/detection_loss‚0.7135458145211526 /loss/FRt1/detection_loss‚2.6949672576259163 1loss/FBt1/assignment_loss‚0.05263830107287504 0loss/FBt2/assignment_loss‚0.2108372761802882 0loss/FBt4/assignment_loss‚0.5110801290720657 0loss/SRqqt1/detection_loss‚2.478089225292206 1loss/SRqqt2/detection_loss‚1.4672861192907607 epoch‚3 0loss/FRt3/assignment_loss‚3.9801942884922026ð{½ü“î‚(:ˆ †þåÔ™ÂÓÇ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚188.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r÷Š`Yï‚(:Î †þåÔ¾¦ëÉ disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 proc.memory.rssMB‚2725.875 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda1.in‚0% disk./.usageGB‚295.86083221435547 disk.dm-0.in‚0.078125 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.rbd1.out‚ 29.1962890625 cpu‚25.263196583393583 disk.drbd2265.in‚0 network.sent‚ 1.0916897e+07 disk.sda.out‚ 156.46484375 disk.sda3.in‚0& disk.nvme0n1.out‚17235.87255859375# disk.dm-0.out‚17235.87255859375 disk.sda1.out‚0* disk./.usagePercent‚19.855468693678343 disk.sda2.out‚0 disk.sda3.out‚ 156.46484375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 network.recv‚ 1.513174e+06 disk.sda2.in‚0 disk.dm-1.out‚ 156.46484375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚95.7937479019165* proc.memory.percent‚16.637420654296875( disk.nvme0c0n1.out‚17235.87255859375 disk.drbd2080.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 689.15234375ÂU˜ÿ“ð‚(:ˆ •þåÔÍ’¹Ç gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚186.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ1]Vñ‚(:Ë •þåÔ¬ÁË" disk.dm-0.out‚17290.9404296875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 689.39453125 proc.memory.rssMB‚2725.875* proc.memory.percent‚16.637420654296875 disk.drbd1716.in‚0 disk.drbd2080.in‚0) disk./.usagePercent‚19.85707155685022 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚17290.9404296875 disk.sda.out‚ 157.28515625 disk.sda3.out‚ 157.28515625 disk.dm-1.in‚0 disk.dm-1.out‚ 157.28515625$ memory_percent‚95.79226970672607 cpu‚25.262410689776566 proc.cpu.threads‚17 network.sent‚ 1.1014129e+07 network.recv‚ 1.518281e+06 disk.sda2.out‚0 disk.nvme0n1.in‚0.078125% disk.nvme0n1.out‚17290.9404296875 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd2265.out‚0% disk./.usageGB‚295.88471603393555 disk.drbd2266.out‚0 disk.rbd1.out‚ 29.1962890625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.rbd0.out‚0{M°2ò- _wandbruntime‚4168  _runtime‚4168ö.ñšÉóà % _timestamp‚1.7884444514252996e+09 _runtime‚4168.472625283 _step‚369 ' lr-AdamW/pg1‚0.00012091873831953055 ' lr-AdamW/pg2‚0.00012091873831953055 trainer/global_step‚4549ÍÄÚ¯ÎôÈ ' lr-AdamW/pg1‚0.00012091873831953055 ' lr-AdamW/pg2‚0.00012091873831953055 trainer/global_step‚4549 % _timestamp‚1.7884444514252996e+09 _runtime‚4168.472625283 _step‚369ñ€Œ|ª2õ- _wandbruntime‚4168  _runtime‚41683% ûöëO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325116455377920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80018970 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4520132616192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178862037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4294588157440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70831395 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353088692125696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312122811593728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4295371282944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320821884390912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320821884390912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70831921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚370 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98843067 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320821867220480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4278159894690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312122828633088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.alÌ“öl.allocated‚357608824741888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278941095454 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325117255673856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325117255673856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320821867220480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98866177 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178885149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47051314 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23780607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23780084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357608824741888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278941095454 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325117255673856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325117255673856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320821867220480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98866177 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178885149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47051314 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23780607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23780084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325116455377920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80018970 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4520132616192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178862037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4294588157440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70831395 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353088692125696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312122811593728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4295371282944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320821884390912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320821884390912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70831921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0òÒsPÆ2ø- _wandbruntime‚4169  _runtime‚4169¬Ùÿ³ü&ùöM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316469511474082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178900411 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80036010 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23785620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325186014436864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70846384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279865052066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80036010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4296298781696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70846926 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320890498793984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353164374316032 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312189629382656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23785081 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357685487523328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884444522391253e+09 _step‚371 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80036012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98887537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212615168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98887537 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178900411 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80036012 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312189646422016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4295515642880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4520925364736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 _runtime‚4169.286827086 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178923549 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47061306 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47061303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4295515642880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320890498793984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4521113207296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187842560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325186814746112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320890515964416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4296298781696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357685274908160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316468713233750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32Yêßé'úûM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47061306 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47061303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4295515642880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320890498793984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4521113207296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187842560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325186814746112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320890515964416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4296298781696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357685274908160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316468713233750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316469511474082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178900411 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80036010 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23785620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325186014436864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70846384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279865052066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80036010 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4296298781696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70846926 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320890498793984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353164374316032 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312189629382656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23785081 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357685487523328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884444522391253e+09 _step‚371 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80036012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98887537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212615168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98887537 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178900411 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80036012 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312189646422016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4295515642880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4520925364736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 _runtime‚4169.286827086 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178923549 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010ó¦ÉŒ2û- _wandbruntime‚4169  _runtime‚4169E¸Ý‚üÑ _step‚372 $loss/total_loss‚49.7246315044097 0loss/FRt2/assignment_loss‚2.6743724291974846 2loss/SRqqt2/assig’Â,÷Õnment_loss‚1.6318192351155165 2loss/SRqqt4/assignment_loss‚1.9493550141173672 0loss/FBt1/assignment_loss‚0.9267764091187668 /loss/FRt2/detection_loss‚1.5747718050737272 .loss/FRt3/detection_loss‚2.825875873074812 .loss/FRt1/detection_loss‚2.374867313851913 /loss/FRt3/assignment_loss‚4.095333527116215 /loss/FBt2/detection_loss‚0.6198544040126627 _runtime‚4169.287983489 0loss/SRqqt4/detection_loss‚1.424688582185162 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚4.089112463096778 /loss/FRt4/assignment_loss‚3.267045729926654 0loss/FBt3/assignment_loss‚0.6700010763274299 1loss/SRqqt3/detection_loss‚2.5739798567913197 .loss/FBt3/detection_loss‚2.560971200466156 /loss/FBt4/detection_loss‚3.3063400089740753 % _timestamp‚1.7884444522408366e+09 2loss/SRqqt1/assignment_loss‚1.7230825169092936 1loss/FBt2/assignment_loss‚0.24256023135598123 0loss/FBt4/assignment_loss‚0.7213282535473505 /loss/FBt1/detection_loss‚2.7309944799968173 2loss/SRqqt3/assignment_loss‚2.0208737005790076 .loss/FRt4/detection_loss‚2.129143969288894 1loss/SRqqt1/detection_loss‚2.1564083760199337 1loss/SRqqt2/detection_loss‚1.4350750482663877 epoch‚3›Ë:\ýÖ .loss/FBt3/detection_loss‚2.560971200466156 /loss/FBt4/detection_loss‚3.3063400089740753 % _timestamp‚1.7884444522408366e+09 2loss/SRqqt1/assignment_loss‚1.7230825169092936 1loss/FBt2/assignment_loss‚0.24256023135598123 0loss/FBt4/assignment_loss‚0.7213282535473505 /loss/FBt1/detection_loss‚2.7309944799968173 2loss/SRqqt3/assignment_loss‚2.0208737005790076 .loss/FRt4/detection_loss‚2.129143969288894 1loss/SRqqt1/detection_loss‚2.1564083760199337 1loss/SRqqt2/detection_loss‚1.4350750482663877 epoch‚3 _step‚372 $loss/total_loss‚49.7246315044097 0loss/FRt2/assignment_loss‚2.6743724291974846 2loss/SRqqt2/assignment_loss‚1.6318192351155165 2loss/SRqqt4/assignment_loss‚1.9493550141173672 0loss/FBt1/assignment_loss‚0.9267764091187668 /loss/FRt2/detection_loss‚1.5747718050737272 .loss/FRt3/detection_loss‚2.825875873074812 .loss/FRt1/detection_loss‚2.374867313851913 /loss/FRt3/assignment_loss‚4.095333527116215 /loss/FBt2/detection_loss‚0.6198544040126627 _runtime‚4169.287983489 0loss/SRqqt4/detection_loss‚1.424688582185162 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚4.089112463096778 /loss/FRt4/assignment_loss‚3.267045729926654 0loss/FBt3/assignment_loss‚0.6700010763274299 1loss/SRqqt3/detection_loss‚2.5739798567913197ôþö$&“þ‚(:ˆ ¤þåÔŽ¯Ç gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.078' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.4156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜeéJÿ‚(:¿ ¤þåÔåý¢Ê disk.sda.in‚0 proc.memory.rssMB‚2725.875 disk.drbd2080.out‚0 disk.rbd1.in‚0 cpu‚25.26166702255214 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚17334.970703125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda.out‚ 157.734375 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.79362869262695* proc.memory.percent‚16.637420654296875* disk./.usagePercent‚19.857667286382398 disk.rbd1.out‚ 29.2587890625& disk.nvme0c0n1.out‚17334.970703125 proc.cpu.threads‚17% disk./.usageGB‚295.89359283447266 disk.sda3.in‚0 disk.sda3.out‚ 157.734375! disk.dm-0.out‚17334.970703125 disk.drbd2080.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 689.171875 disk.dm-1.out‚ 157.734375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.1020157e+07 network.recv‚ 1.52251e+06¥ˆ™€‚(:’ ³þåÔ€‘¯Ç gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.11' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.422000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kmÞV‚(:Ë ³þåÔ¿´ÏÉ( disk./.usagePercent‚19.8582520075906% disk./.usageGB‚295.90230560302734 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0* proc.memory.percent‚16.637420654296875 disk.dm-1.out‚ 158.29296875 disk.drbd2266.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.1128477e+07 disk.sda2.in‚0$ memory_percent‚95.79267501831055 proc.memory.rssMB‚2725.875 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125( disk.nvme0c0n1.out‚17402.37353515625 disk.sda.out‚ 158.29296875& disk.nvme0n1.out‚17402.37353515625 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 158.29296875& proc.memory.availableMB‚ 689.328125 cpu‚25.260921123493752 disk.drbd2266.out‚0 network.recv‚ 1.528824e+06 disk.drbd2080.in‚0 disk.rbd1.out‚ 29.2587890625 proc.cpu.threads‚17# disk.dm-0.out‚17402.37353515625 disk.sda.in‚0á ?~ž‚‚(:“ ÂþåÔÙÒµÇ gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.897' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.179399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ó=Tƒ‚(:É ÂþåÔœ¨ÜÉ disk.drbd1716.in‚0& proc.memory.availableMB‚ 689.078125 cpu‚25.26017119576048 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚0.078125& disk.nvme0n1.out‚17441.71337890625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda.out‚ 159.05078125 disk.dm-1.out‚ 159.05078125 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚17441.71337890625 disk.sda2.out‚0# memory_percent‚95.7942008972168 network.sent‚ 1.1134734e+07 disk.nvme0n1.in‚0.078125 disk.drbd2080.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚19.858915067104284 disk.dm-0.in‚0.078125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 29.416015625$ disk./.usageGB‚295.9121856689453 disk.drbd2266.in‚0 disk.sda3.in‚0# disk.dm-0.out‚17441.71337890625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚2725.875* proc.memory.percent‚16.637420654296875 network.recv‚ 1.533176e+06 disk.sda3.out‚ 159.05078125 disk.dm-1.in‚0Ï2„- _wandbruntime‚4212  _runtime‚4212Na nÉ…Ã % _timestamp‚1.7884444956622531e+09 _runtime‚4212.709468447 _step‚373 ' lr-AdamW/pg1‚0.00012000060415082286 ' lr-AdamW/pg2‚0.00012000060415082286 trainer/global_step‚4599ÖHÀÎ†È trainer/global_step‚4599 % _timestamp‚1.7884444956622531e+09 _runtime‚4212.709468447 _step‚373 ' lr-AdamW/pg1‚0.00012000060415082286 ' lr-AdamW/pg2‚0.00012000060415082286õlk‰…2‡- _wandbruntime‚4212  _runtime‚4212;vÖ ò'ˆìO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99913119 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4341759968768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328594422738944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180807201 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80870972 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71582537 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4568789632000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47550911 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361441777033216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4325152506862 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315463701040128 trainer/global_step‚4599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80870972 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4341759968768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4340976843264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185758720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99913119 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328594422738944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4340976843264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47550914 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356872776869888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4568975390720ÁOì÷'‰ñO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185758720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99913119 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328594422738944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4340976843264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47550914 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356872776869888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4568975390720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99913119 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4341759968768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328594422738944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180807201 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80870972 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71582537 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4568789632000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47550911 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361441777033216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4325152506862 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315463701040128 trainer/global_step‚4599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80870972 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4341759968768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4340976843264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328ö>”¥é2Š- _wandbruntime‚4213  _runtime‚4213M ɇ‹öM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4326076463474 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99934453 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324322077469184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4341904328704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324322094639616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361518216608256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47560906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4342687467520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 FDeviceStatsMonito-æw-ur.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4569757548032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180845601 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180845601 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4213.532644624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4342687467520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99934453 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319855814091558 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328664782107136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180822463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 trainer/global_step‚4599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80888012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71597492 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80888010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99957589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71597492 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80888010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99957589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4569757548032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223100928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180845601 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180845601 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4213.532644624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4342687467520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99934453 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319855814091558 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328664782107136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180822463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 trainer/global_step‚4599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80888012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72333819 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4371363918270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332072390099968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24283305 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72333291 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182706141 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81722970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48050514 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚318804607525888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100983171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332072390099968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72333819 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4371363918270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332072390099968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24283305 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72333291 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182706141 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81722970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48050514 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚318804607525888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100983171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332072390099968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182767653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360732593349632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4618919494656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332141949158912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332142749468160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318871408275456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327753656144384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323243713189698 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101004505 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4389076153344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81740010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332142749468160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101027641 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81740012 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101027641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327753673314816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323242914949366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚365351298131968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72348287 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327753656144384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81740012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4389076153344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48060506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4618729554944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101004505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360732568577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4257.699366412 _step‚379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24288320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365351512844288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81740010 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72348826 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318871408275456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327753656144384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323243713189698 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101004505 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4389076153344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81740010 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332142749468160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101027641 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81740012 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101027641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327753673314816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323242914949366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚365351298131968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72348287 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327753656144384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81740012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4389076153344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48060506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4618729554944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101004505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360732568577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4257.699366412 _step‚379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24288320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365351512844288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81740010 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72348826 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182767653 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214712320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360732593349632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031û ?¤‘2Ÿ- _wandbruntime‚4257  _runtime‚4257:ó~lW Ñ /loss/FRt4/assignment_loss‚3.222803628912159 /loss/FRt3/assignment_loss‚4.527694016695023 0loss/FBt3/assignment_loss‚0.5914590059281863 % _timestamp‚1.7884445406534245e+09 2loss/SRqqt3/assignment_loss‚2.4099726509302855 2loss/SRqqt4/assignment_loss‚1.8198764311927784 /loss/FRt4/detection_loss‚2.1441269744844997 /loss/FRt1/assignment_loss‚4.438366422057152 0loss/FBt1/assignment_loss‚0.6097488224506378 0loss/FBt4/assignment_loss‚0.3619779699568945 /loss/FRt1/detection_loss‚2.5144545517861845 1loss/SRqqt2/detection_loss‚1.4057961303640056 /loss/FBt3/detection_loss‚2.2929498652617135 2loss/SRqqt1/assignment_loss‚2.1559337255986115 0loss/FBt2/assignment_loss‚0.1767502169733265 /loss/FRt3/detection_loss‚2.8370093256235123 1loss/SRqqt1/detection_loss‚2.4039921666446484 1loss/SRqqt4/detection_loss‚1.5269071502210219 epoch‚3 _runtime‚4257.700585133 0loss/SRqqt3/detection_loss‚2.534899078309536 %loss/total_loss‚50.82446976670887 _step‚380 /loss/FRt2/assignment_loss‚3.123405542779476 /loss/FRt2/detection_loss‚1.8091273323652592 .loss/FBt1/detection_loss‚2.616086220741272 1loss/SRqqt2/assignment_loss‚1.330130305241894 .loss/FBt2/detection_loss‚0.702735230766282 /loss/FBt4/detection_loss‚3.2682670014245168 trainer/global_step‚4649«0ü˜¡Ö /loss/FRt2/assignment_loss‚3.123405542779476 /loss/FRt2/detection_loss‚1.8091273323652592 .loss/FBt1/detection_loss‚2.616086220741272#¶9Ä 1loss/SRqqt2/assignment_loss‚1.330130305241894 .loss/FBt2/detection_loss‚0.702735230766282 /loss/FBt4/detection_loss‚3.2682670014245168 trainer/global_step‚4649 /loss/FRt4/assignment_loss‚3.222803628912159 /loss/FRt3/assignment_loss‚4.527694016695023 0loss/FBt3/assignment_loss‚0.5914590059281863 % _timestamp‚1.7884445406534245e+09 2loss/SRqqt3/assignment_loss‚2.4099726509302855 2loss/SRqqt4/assignment_loss‚1.8198764311927784 /loss/FRt4/detection_loss‚2.1441269744844997 /loss/FRt1/assignment_loss‚4.438366422057152 0loss/FBt1/assignment_loss‚0.6097488224506378 0loss/FBt4/assignment_loss‚0.3619779699568945 /loss/FRt1/detection_loss‚2.5144545517861845 1loss/SRqqt2/detection_loss‚1.4057961303640056 /loss/FBt3/detection_loss‚2.2929498652617135 2loss/SRqqt1/assignment_loss‚2.1559337255986115 0loss/FBt2/assignment_loss‚0.1767502169733265 /loss/FRt3/detection_loss‚2.8370093256235123 1loss/SRqqt1/detection_loss‚2.4039921666446484 1loss/SRqqt4/detection_loss‚1.5269071502210219 epoch‚3 _runtime‚4257.700585133 0loss/SRqqt3/detection_loss‚2.534899078309536 %loss/total_loss‚50.82446976670887 _step‚380ü¼>“¢‚(:ˆ þþåÔ½»åÇ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ4¿-N£‚(:à þþåÔ•ÇÍÊ network.recv‚ 1.551158e+06% disk./.usageGB‚295.93527603149414 disk.sda2.out‚0* proc.memory.percent‚16.637420654296875 network.sent‚ 1.1260425e+07 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0* disk./.usagePercent‚19.860464680709043 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 29.7451171875 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚17645.6767578125 disk.sda.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 161.28125 disk.dm-1.out‚ 161.28125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚0.078125 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0$ memory_percent‚95.79651355743408( proc.memory.availableMB‚ 688.69921875 cpu‚25.257218352337077% disk.nvme0n1.out‚17645.6767578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 161.28125 proc.memory.rssMB‚2725.875" disk.dm-0.out‚17645.6767578125 disk.drbd2265.in‚0 disk.dm-1.in‚0ñ)§“¤‚(:ˆ ÿåÔšÑºÇ gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êg UK¥‚(:À ÿåÔÀõØÊ disk.dm-1.out‚161.875 network.sent‚ 1.1368139e+07 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.078125 cpu‚25.256436419593562 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 29.7451171875 disk.sda1.out‚0( disk.nvme0c0n1.out‚17703.53076171875* disk./.usagePercent‚19.861168445420667 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚95.79708576202393% disk./.usageGB‚295.94576263427734 disk.sda1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.637420654296875 proc.cpu.threads‚17# disk.dm-0.out‚17703.53076171875 disk.rbd0.in‚0 disk.sda3.out‚161.875& disk.nvme0n1.out‚17703.53076171875 network.recv‚ 1.556763e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0.078125 disk.sda.out‚161.875( proc.memory.availableMB‚ 688.60546875 proc.memory.rssMB‚2725.875æ§%Œ“¦‚(:ˆ œÿåÔêï¤Ç gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÈ›®P§‚(:Å œÿåÔ‹û»Ê$ disk./.usageGB‚295.9548530578613 disk.sda1.in‚0$ memory_percent‚95.79553604125977& proc.memory.availableMB‚ 688.859375 proc.memory.rssMB‚2725.875 disk.sda3.out‚ 162.328125 disk.dm-1.out‚ 162.328125 disk.drbd2080.in‚0* disk./.usagePercent‚19.861778511374766 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚17747.79638671875 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.out‚0* proc.memory.percent‚16.637420654296875 network.recv‚ 1.560347e+06 disk.rbd1.out‚ 29.8076171875 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0& disk.nvme0n1.out‚17747.79638671875 disk.drbd2080.out‚0 cpu‚25.255779640119275 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 162.328125 disk.dm-1.in‚0 disk.dm-0.in‚0.078125 network.sent‚ 1.137327e+07( disk.nvme0c0n1.out‚17747.79638671875 disk.drbd2265.in‚0jÙr2¨- _wandbruntime‚4301  _runtime‚4301u‰'Ç©Á _step‚381 & lr-AdamW/pg1‚0.0001181376577772059 & lr-AdamW/pg2‚0.0001181376577772059 trainer/global_step‚4699 % _timestamp‚1.7884445842190456e+09 _runtime‚4301.266329076ŽŒe‰ÌªÆ & lr-AdamW/pg1‚0.0001181376577772059 & lr-AdamW/pg2‚0.0001181376577772059 trainer/global_step‚4699 % _timestamp‚1.7884445842190456e+09 _runtime‚4301.266329076 _step‚381ý®82«- _wandbruntime‚4301  _runtime‚4301‰NÍXõ'¬ïO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184628193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335551157756928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82574972 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322145496972288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82574972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24535484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335550357460992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369107927388672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326562274061842 $ _timestamp‚1.788444584220034e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331116620416512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 _step‚382 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102076333 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331116620416512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4434537340416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102076333 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326563072301966 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4434537340416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82574970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184651305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82574970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512@>ú'­ôO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322145496972288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82574972 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24535484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335550357460992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369107927388672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326562274061842 $ _timestamp‚1.788444584220034e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331116620416512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 _step‚382 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102076333 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331116620416512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4434537340416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102076333 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326563072301966 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4434537340416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82574970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184651305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192050176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82574970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184628193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335551157756928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82574972þò+Ñ2®- _wandbruntime‚4302  _runtime‚4302“w_¯ùM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364516678093824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4434681700352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 _runtime‚ 4302.10588625 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335619916519936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4435464839168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82592012 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335620716829184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48560106 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4435464839168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326630814047506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4667880027136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326630015807174 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4417718085318 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4434681700352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82592010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4667690087424 % _timestamp‚1.7884445850578976e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102074557 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322212297721856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4418499286290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184689705 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102074557 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322212314761216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369184582893568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369184368181248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73100062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48560103 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331185251990016 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364516702866432 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331185234819584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82592010 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73100593 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102097693 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24539959 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _step‚383 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184689705 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82592012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136b ½î'°þM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24539959 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 _step‚383 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184689705 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82592012 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364516678093824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4434681700352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 _runtime‚ 4302.10588625 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335619916519936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4435464839168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82592012 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335620716829184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48560106 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4435464839168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326630814047506 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4667880027136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326630015807174 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4417718085318 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4434681700352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82592010 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4667690087424 % _timestamp‚1.7884445850578976e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102074557 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322212297721856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4418499286290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184689705 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102074557 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322212314761216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369184582893568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369184368181248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73100062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48560103 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331185251990016 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364516702866432 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331185234819584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82592010 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73100593 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102097693ÿj[!ò2±- _wandbruntime‚4302  _runtime‚4302GÿÈ?P²Ê 0loss/SRqqt4/detection_loss‚1.331639673155935 _runtime‚4302.106869676 2loss/SRqqt3/assignment_loss‚1.9489903450012207 -loss/FRt3/detection_loss‚2.66249106569988 0loss/SRqqt2/detection_loss‚1.532412560449706 trainer/global_step‚4699 .loss/FRt1/detection_loss‚2.657076734304428 .loss/FRt2/detection_loss‚2.559035344980657 0loss/SRqqt3/detection_loss‚2.488364590331912 2loss/SRqqt2/assignment_loss‚1.3708156153766646 2loss/SRqqt4/assignment_loss‚1.8374240800090458 0loss/FBt1/assignment_loss‚0.4003748754391078 0loss/FBt2/assignment_loss‚0.2667811837991401 .loss/FRt1/assignment_loss‚3.68692846596241 .loss/FBt1/detection_loss‚4.462971031665802 $loss/total_loss‚50.5278053359924 % _timestamp‚1.7884445850596242e+09 0loss/FBt3/assignment_loss‚0.4358599791691328 0loss/FRt2/assignment_loss‚3.0405169442296027 0loss/FRt3/assignment_loss‚3.9703987109951857 .loss/FRt4/detection_loss‚2.160988588048064 epoch‚3 _step‚384 0loss/FRt4/assignment_loss‚2.9959181262099226 /loss/FBt4/assignment_loss‚0.359354048302131 1loss/SRqqt1/detection_loss‚2.2010040754495663 /loss/FBt3/detection_loss‚2.3594297021627426 2loss/SRqqt1/assignment_loss‚1.7961605932823448 /loss/FBt2/detection_loss‚0.5905513087343006 -loss/FBt4/detection_loss‚3.41231769323349Â0U³Ï 2loss/SRqqt3/assignment_loss‚1.9489903450012207 -loss/FRt3/detection_loss‚2.66249106569988 0loss/SRqqt2/detection_loss‚1.532412560449706 trainer/global_step‚4699 .loss/FRt1/detection_loss‚2.657076734304428 .loss/FRt2/detection_loss‚2.559035344980657 0loss/SRqqt3/detection_loss‚2.488364590331912 2loss/SRqqt2/assignment_loss‚1.3708156153766646 2loss/SRqqt4/assignment_loss‚1.8374240800090458 0loss/FBt1/assignment_loss‚0.4003748754391078 0loss/FBt2/assignment_loss‚0.2667811837991401 .loss/FRt1/assignment_loss‚3.68692846596241 .loss/FBt1/detection_loss‚4.462971031665802 $loss/total_loss‚50.5278053359924 % _timestamp‚1.7884445850596242e+09 0loss/FBt3/assignment_loss‚0.4358599791691328 0loss/FRt2/assignment_loss‚3.0405169442296027 0loss/FRt3/assignment_loss‚3.9703987109951857 .loss/FRt4/detection_loss‚2.160988588048064 epoch‚3 _step‚384 0loss/FRt4/assignment_loss‚2.9959181262099226 /loss/FBt4/assignment_loss‚0.359354048302131 1loss/SRqqt1/detection_loss‚2.2010040754495663 /loss/FBt3/detection_loss‚2.3594297021627426 2loss/SRqqt1/assignment_loss‚1.7961605932823448 /loss/FBt2/detection_loss‚0.5905513087343006 -loss/FBt4/detection_loss‚3.41231769323349 0loss/SRqqt4/detection_loss‚1.331639673155935 _runtime‚4302.106869676€x°éú’´‚(:‡ «ÿåÔ€–ÉÇ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù!/ŒTµ‚(:É «ÿåÔúãÉÊ% disk./.usageGB‚295.96959686279297 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚17787.1240234375* proc.memory.percent‚16.637420654296875 network.sent‚ 1.1379092e+07 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 162.8515625 disk.sda3.out‚ 162.8515625 disk.rbd1.in‚0 disk.rbd1.out‚ 29.9873046875 cpu‚25.255065776621024 disk.sda1.in‚0 disk.drbd2266.out‚0$ memory_percent‚95.78855037689209( proc.memory.availableMB‚ 690.00390625 proc.memory.rssMB‚2725.875 disk.dm-1.out‚ 162.8515625 disk.dm-0.in‚0.078125 disk.drbd2080.out‚0 proc.cpu.threads‚17 network.recv‚ 1.564355e+06 disk.sda.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚19.862767980494603 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.078125' disk.nvme0c0n1.out‚17787.1240234375% disk.nvme0n1.out‚17787.1240234375 disk.drbd2080.in‚0 @h“¶‚(:ˆ ºÿåÔ｡Ç gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚191.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.3964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†N×=·‚(:² ºÿåÔÆ—ÍÊ disk.dm-0.in‚0.078125 disk.dm-0.out‚ 17831.5859375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.254357693465302 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 17831.5859375 proc.memory.rssMB‚2725.875* proc.memory.percent‚16.637420654296875! disk./.usageGB‚295.9814453125 disk.drbd1716.in‚0 network.sent‚ 1.148793e+07 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 163.390625 disk.dm-1.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚0$ memory_percent‚95.79010009765625" proc.memory.availableMB‚689.75 network.recv‚ 1.570775e+06 disk.sda2.out‚0 disk.sda3.out‚ 163.390625 disk.dm-1.out‚ 163.390625" disk.nvme0n1.out‚ 17831.5859375 disk.rbd1.out‚ 30.2763671875 proc.cpu.threads‚17* disk./.usagePercent‚19.863563139895952 disk.nvme0c0n1.in‚0.078125õ ®M‘¸‚(:† ÉÿåÔêì×Ç gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚130.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( ´W¹‚(:Ì ÉÿåÔ–îàÊ disk.rbd1.out‚ 30.4560546875$ memory_percent‚95.78855037689209 proc.memory.rssMB‚2725.875( disk.nvme0c0n1.out‚17897.28271484375 disk.sda3.out‚ 164.04296875# disk.dm-0.out‚17897.28271484375 disk.drbd2265.out‚0 disk.rbd0.out‚0* disk./.usagePercent‚19.864287897225385 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.dm-1.out‚ 164.04296875 disk.nvme0c0n1.in‚0.078125 disk.drbd1716.out‚0& disk.nvme0n1.out‚17897.28271484375 cpu‚25.25365262772778# disk./.usageGB‚295.992244720459 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.637420654296875 disk.drbd2080.out‚0 network.recv‚ 1.574249e+06 disk.sda.out‚ 164.04296875 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.1493323e+07 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2265.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 690.00390625³ê2º- _wandbruntime‚4345  _runtime‚4345ت¨É»Ã % _timestamp‚1.7884446285076494e+09 _runtime‚4345.554894687 _step‚385 ' lr-AdamW/pg1‚0.00011719323380752864 ' lr-AdamW/pg2‚0.00011719323380752864 trainer/global_step‚4749÷ayÎ¼È ' lr-AdamW/pg1‚0.00011719323380752864 ' lr-AdamW/pg2‚0.00011719323380752864 trainer/global_step‚4749 % _timestamp‚1.7884446285076494e+09 _runtime‚4345.554894687 _step‚385tr€ 2½- _wandbruntime‚4345  _runtime‚4345¢íTö'¾ðO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339028324822016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372941185330688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83426970 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103123275 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103123275 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339029125117952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49049714 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372940978993664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339028324822016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83426972 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73836997 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4463786741086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4480142900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334548199091712 _step‚386 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103146385 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368225130192896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4715873573376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339029125117952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368225105420288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186550245 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186573357 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚334548199091712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325486386418688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚4345.555848377 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337024ýÔ´9¿õO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103123275 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339029125117952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49049714 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372940978993664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.£2ÝÂ%large_pool.current‚24772608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339028324822016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83426972 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73836997 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4463786741086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4480142900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334548199091712 _step‚386 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103146385 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368225130192896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4715873573376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339029125117952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368225105420288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186550245 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186573357 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚334548199091712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325486386418688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 _runtime‚4345.555848377 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206337024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339028324822016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372941185330688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83426970 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103123275 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521‚wƒ±?2À- _wandbruntime‚4346  _runtime‚4346®Û Û'ÁúM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368300812383232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334616813494784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330017914905314 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4481070386176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83444010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186588619 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334616830665216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4481853524992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334616813494784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4464710697698 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186588619 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 _runtime‚4346.393853234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4481070386176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49059706 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339098684190208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339097883880960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73851941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49059703 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339098684190208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83444012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73851430 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373017845391360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330017116664982 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103144609 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4717033008128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186611757 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83444010 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368300787610624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325553204207616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334616830665216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚387 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186611757 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83444010 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368300787610624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325553204207616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334616830665216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 _step‚387 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4481070386176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49059706 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339098684190208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339097883880960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73851941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49059703 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339098684190208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83444012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73851430 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373017845391360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330017116664982 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103144609 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4717033008128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865ƒô(u2Ã- _wandbruntime‚4346  _runtime‚4346®ÜžSÄÍ trainer/global_step‚4749 % _timestamp‚1.7884446293478699e+09 0loss/FRt1/assignment_loss‚3.7794599692995954 /loss/FRt4/assignment_loss‚3.215970713080782 2loss/SRqqt1/assignment_loss‚1.4261693277706702 0loss/FBt4/assignment_loss‚0.5543650537648708 /loss/FBt2/detection_loss‚0.5590946941226543 .loss/FRt2/detection_loss‚1.887435323581463 1loss/SRqqt2/detection_loss‚1.3806907534599304 /loss/FRt2/assignment_loss‚2.437435060012631 1loss/SRqqt3/assignment_loss‚2.428490300774574 0loss/FBt2/assignment_loss‚0.2509165997630301 /loss/FRt4/detection_loss‚1.8195443026947253 .loss/FBt4/detection_loss‚3.510117449930736 0loss/FBt3/assignment_loss‚0.5816618827488559 .loss/FRt3/detection_loss‚2.986142414231454 1loss/SRqqt3/detection_loss‚2.6450856566429137 _runtime‚4346.395021519 _step‚388 -loss/FRt1/detection_loss‚2.49002173615665 0loss/SRqqt1/detection_loss‚2.288469119866689 %loss/total_loss‚50.08081721700612 /loss/FRt3/assignment_loss‚4.927396051345333 2loss/SRqqt2/assignment_loss‚1.4912035411526992 2loss/SRqqt4/assignment_loss‚1.8983608131554532 -loss/FBt3/detection_loss‚2.53766803992422 0loss/FBt1/assignment_loss‚0.4536946468846509 1loss/SRqqt4/detection_loss‚1.2520991754817201 epoch‚3 /loss/FBt1/detection_loss‚3.2793245911598206gÇÂXÅÒ 1loss/SRqqt3/detection_loss‚2.6450856566429137 _runtime‚4346.395021519 _step‚388 -loss/FRt1/detection_loss‚2.49002173615665 0loss/SRqqt1/detection_loss‚2.288469119866689 %loss/total_loss‚50.08081721700612 /loss/FRt3/assignment_loss‚4.927396051345333 2loss/SRqqt2/assignment_loss‚1.4912035411526992 2loss/SRqqt4/assignment_loss‚1.8983608131554532 -loss/FBt3/detection_loss‚2.53766803992422 0loss/FBt1/assignment_loss‚0.4536946468846509 1loss/SRqqt4/detection_loss‚1.2520991754817201 epoch‚3 /loss/FBt1/detection_loss‚3.2793245911598206 trainer/global_step‚4749 % _timestamp‚1.7884446293478699e+09 0loss/FRt1/assignment_loss‚3.7794599692995954 /loss/FRt4/assignment_loss‚3.215970713080782 2loss/SRqqt1/assignment_loss‚1.4261693277706702 0loss/FBt4/assignment_loss‚0.5543650537648708 /loss/FBt2/detection_loss‚0.5590946941226543 .loss/FRt2/detection_loss‚1.887435323581463 1loss/SRqqt2/detection_loss‚1.3806907534599304 /loss/FRt2/assignment_loss‚2.437435060012631 1loss/SRqqt3/assignment_loss‚2.428490300774574 0loss/FBt2/assignment_loss‚0.2509165997630301 /loss/FRt4/detection_loss‚1.8195443026947253 .loss/FBt4/detection_loss‚3.510117449930736 0loss/FBt3/assignment_loss‚0.5816618827488559 .loss/FRt3/detection_loss‚2.986142414231454„{÷Ô¨òÆ‚(:ˆ ØÿåÔÕ×ÍÇ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.627' gpu.0.enforcedPowerLimitWatts‚tñ€Ì¡500.0 gpu.0.powerPercent‚36.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø U`RÇ‚(:Ç ØÿåÔûö‰Ê disk.rbd1.out‚ 30.6123046875( disk.nvme0c0n1.out‚17945.81005859375 network.sent‚ 1.1499182e+07 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚0.078125% proc.memory.availableMB‚ 689.96875 cpu‚25.25290572290041 network.recv‚ 1.578518e+06$ disk./.usageGB‚296.0066833496094 disk.drbd2266.out‚0 disk.sda3.out‚ 164.6640625 disk.drbd1716.out‚0* proc.memory.percent‚16.637420654296875 disk.drbd2265.out‚0 disk.sda1.out‚0* disk./.usagePercent‚19.865256885742486 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚17945.81005859375# disk.dm-0.out‚17945.81005859375$ memory_percent‚95.78876495361328 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.dm-1.out‚ 164.6640625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 164.6640625 disk.sda2.out‚0 disk.dm-0.in‚0.078125 proc.memory.rssMB‚2725.875 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0K¸ È‚(:’ çÿåÔ ÏäÇ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚188.88' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.775999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jŒ:(WÉ‚(:Ì çÿåÔ‚àáÉ disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 165.6171875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.recv‚ 1.585103e+06& disk.nvme0n1.out‚18011.07861328125 disk.sda3.in‚0( proc.memory.availableMB‚ 690.18359375 disk.nvme0n1.in‚0.078125 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚2725.875% disk./.usageGB‚296.01445388793945 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚18011.07861328125 disk.sda.in‚0 disk.sda3.out‚ 165.6171875 proc.cpu.threads‚17* disk./.usagePercent‚19.865778373089732 disk.drbd1716.in‚0# disk.dm-0.out‚18011.07861328125 disk.drbd2265.out‚0 cpu‚25.252216286969475 disk.drbd2080.in‚0$ memory_percent‚95.78745365142822* proc.memory.percent‚16.637420654296875 disk.dm-1.out‚ 165.6171875 disk.dm-0.in‚0.078125 disk.rbd1.out‚ 30.6123046875 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda2.out‚0 network.sent‚ 1.1607781e+07 disk.drbd1716.out‚0 disk.drbd2265.in‚0è?ʆ“Ê‚(:ˆ öÿåÔüå¸È gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚38.0 gpu.0.powerWatts‚185.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.1442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ïYË‚(:Î öÿåÔ¹ñ«É disk.nvme0c0n1.in‚0.078125 disk.drbd1716.in‚0 disk.dm-1.out‚ 166.07421875 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18059.48095703125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚95.79050540924072* proc.memory.percent‚16.637420654296875 cpu‚25.251589658795783 network.sent‚ 1.1614143e+07 disk.rbd1.out‚ 30.8154296875 disk.rbd0.out‚0& disk.nvme0n1.out‚18059.48095703125 disk.sda3.out‚ 166.07421875 network.recv‚ 1.589571e+06$ disk./.usageGB‚295.9717826843262# disk.dm-0.out‚18059.48095703125 disk.drbd2080.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚0.078125 disk.sda.out‚ 166.07421875 proc.memory.rssMB‚2725.875* disk./.usagePercent‚19.862914672811723 disk.drbd2266.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 689.68359375š¾2Ì- _wandbruntime‚4390  _runtime‚4390ã*ìYÇÍÁ _runtime‚4390.019869712 _step‚389 & lr-AdamW/pg1‚0.0001162404378094137 & lr-AdamW/pg2‚0.0001162404378094137 trainer/global_step‚4799 % _timestamp‚1.7884446729726686e+09»“ ÌÎÆ _step‚389 & lr-AdamW/pg1‚0.0001162404378094137 & lr-AdamW/pg2‚0.0001162404378094137 trainer/global_step‚4799 % _timestamp‚1.7884446729726686e+09 _runtime‚4390.019869712…lçô2Ï- _wandbruntime‚4390  _runtime‚4390ˆÞêö'ÐðO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4526531586560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4765000952832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188472297 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104193327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25038449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342507092478976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74587211 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚4799 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188495409 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4526531586560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25037900 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376774034325504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4764819388416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342506292183040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84278972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337979760596480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4527314712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104216437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚390 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74587763 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104216437 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333337274017582 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372009214937088 _runtime‚4390.020752408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49549314 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328827258825728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328827275865088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84278972 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84278970 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342506292183040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372009239709696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333336475777458Q¬A‘û'ÑõO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74587763 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104216437 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333337274017582 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372009214937088 _runtime‚4390.020752408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49549314 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328827258825728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328827275865088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84278972 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84278970 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342506292183040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372009239709696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333336475777458 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4526531586560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4765000952832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188472297 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11048832512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104193327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25038449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342507092478976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74587211 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚521 trainer/global_step‚4799 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188495409 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4526531586560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25037900 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376774034325504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4764819388416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342506292183040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84278972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181564416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337979760596480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4527314712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104216437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚390†“Ïð2Ò- _wandbruntime‚4390  _runtime‚4390NŠË¢±ÓùM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84296012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74602699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84296012 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 $ _timestamp‚1.788444673805135e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4765984247296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188510671 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188533809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25042848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 _step‚391 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338048409340416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49559303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74602151 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104237797 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188510671 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84296010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376850906147328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 trainer/global_step‚4799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188533809 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84296012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74602699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84296012 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 $ _timestamp‚1.788444673805135e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4765984247296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188510671 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188533809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25042848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 _step‚391 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338048409340416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49559303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74602151 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 781200972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104237797 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188510671 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84296010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376850906147328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185745408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 trainer/global_step‚4799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188533809 EDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 605072896 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4580124260864 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 190940031 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚85551840 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚85551840 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 585986648 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚347310628766208 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚333395149467648 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 190940031 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚342729916602880 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚521 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 105369908 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0[P0Žb*æÜT \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10672704512 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4580124260864 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1021188096 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 587902464 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚347310628766208 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚50328217 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚24772608 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 302690304 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4562603285638 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 12013121024 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 105388189 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27968 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4580712163328 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1021188096 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚181 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371895808 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4820402874880 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1010 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18283 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 605072896 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3157 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4580124260864 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 190940031 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚85551840 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚85551840 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 585986648 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚347310628766208 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚333395149467648 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 190940031 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚342729916602880 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚521 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 105369908 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 12013121024 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚50328214 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 190921748 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚75790290 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 587902464 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚377114537315328 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚24772608 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4563576647454 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1010 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 605072896 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚50336568 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚377114512542720 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1010 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 _step‚394 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚381935304358400 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚342775017200128 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚347355530401792 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4580513201664 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1031 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 105377690 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 105377690 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚338001511163678 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚333437934516224 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1010 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27968 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚17170432 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4581101104128 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 144003584 _runtime‚4475.092162899 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 105395971 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1021188096 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚342775034370560 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚342775017200128 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚381935473134592 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10672704512 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚521 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3157 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 105395971 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371895808 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4820791815680 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚75790474 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019225937 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚85564950 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587902464 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4580513201664 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚347356135474688 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚25453906 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚181 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4562990660806 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚347356135474688 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚347355530401792 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚25453725 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28217 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚75790290 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 587902464 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚377114537315328 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚24772608 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4563576647454 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1010 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 605072896 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚50336568 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚377114512542720 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1010 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18283 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1010 _step‚394 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 172325376 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚381935304358400 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚521 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚521 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚342775017200128 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚347355530401792 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4580513201664 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1031 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 105377690 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 105377690 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚338001511163678 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚333437934516224 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1010 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27968 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚17170432 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4581101104128 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 144003584 _runtime‚4475.092162899 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3157 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 105395971 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1021188096 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚342775034370560 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚342775017200128 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚381935473134592 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚¹‚³ÊU 10672704512 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚521 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3157 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 105395971 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371895808 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4820791815680 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12259950592 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚75790474 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1019225937 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12259950592 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11234443264 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚85564950 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 587902464 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4580513201664 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚347356135474688 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚25453906 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚181 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4562990660806 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚347356135474688 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚347355530401792 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚25453725 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 B&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08054172247648239 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03275733441114426 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12064436823129654 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_1‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35090020298957825 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11386997252702713 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36669203639030457 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17146049439907074 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.33974358439445496 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822238683700562 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2875676453113556 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 epoch‚3 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3973034620285034 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_2_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 particle/accuracy_2_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4393034279346466 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44333332777023315 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910547137260437 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09567586332559586 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11314688622951508 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_3_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.898796796798706 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28815510869026184 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08891750127077103 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011791383847594261 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3402356803417206 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_5‚0.4317525029182434 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 +jet/accuracy_2_of_4‚0.27981382608413696 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17475007474422455 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4413047730922699 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.39099201560020447 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_2‚0.6714832186698914 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4116441309452057 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_6‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9492486119270325 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15580807626247406 particle/accuracy_3_of_7‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963880777359009 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31206944584846497 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2611345052719116 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6305399537086487 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3573593199253082 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818966031074524 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30614033341407776 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.37731480598449707 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3045019209384918 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09695566445589066 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33528760075569153 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4534491300582886 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.338116854429245 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.288170725107193 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12660819292068481 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887623190879822 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2928254306316376 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14774072170257568 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3333333134651184 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3322916626930237 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_3‚0.19288690388202667 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4881410300731659 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881099462509155 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2101198136806488 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0037830278743058443 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09405897557735443 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.46339288353919983 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 particle/accuracy_5_of_7‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4262690246105194 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666913032532 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279868125916 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0090773431584239 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09465614706277847 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0730452686548233 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0900377407670021 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14460375905036926 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19443435966968536 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12361437827348709 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12001486122608185 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39756590127944946 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_4_of_7‚0 _step‚395 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.436507910490036 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2947089672088623 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5301971435546875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07650607079267502 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.006194940768182278 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15051838755607605 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *jet/accuracy_1_of_6‚0.9615384340286255 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.412241667509079 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010107816196978092 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26875001192092896 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7742425203323364 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3849570155143738 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35956788063049316 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10481394827365875 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1361110955476761 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39032116532325745 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 'particle/f_score‚0.3435252904891968 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3588513433933258 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014545678161084652 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3793613314628601 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39990660548210144 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.163059264421463 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚50.22644531069847 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3322724401950836 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.46694645285606384 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4694444239139557 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_1‚0.4181266725063324 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899292588233948 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979872107505798 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013287810608744621 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3057556748390198 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13879527151584625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3864942491054535 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07333333045244217 particle/accuracy_6_of_6‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36116939783096313 particle/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02767537347972393 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5253136157989502 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9862969517707825 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26969268918037415 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20896799862384796 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15859584510326385 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.31308698654174805 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3336600959300995 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2244795858860016 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39058879017829895 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24537040293216705 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896868467330933 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4574255347251892 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6655310988426208 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818370521068573 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14396552741527557 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0891065001487732 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884447715098064e+09 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3167289197444916 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34162840247154236 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39990660548210144 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02440476231276989 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9837095737457275 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14760740101337433 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07375479489564896 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15054979920387268 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06895040720701218 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09705746918916702 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07112908363342285 ,particle/sensitivity‚0.26437267661094666 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10021157562732697 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3662037253379822 jet/accuracy_3_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3686080873012543 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 'particle/accuracy‚0.842323899269104 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14451861381530762 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.32364338636398315 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012599708512425423 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.27812495827674866 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqÇ„~!qt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13139815628528595 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3360566198825836 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 -jet/accuracy_4_of_4‚0.0057404013350605965 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 _runtime‚4488.560030004 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_5_of_6‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09668289870023727 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3579123318195343 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.05468036234378815 jet/accuracy_5_of_5‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5191912651062012 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3824882507324219 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005376344081014395 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3914516866207123 -"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.349086731672287 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16872581839561462 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41837137937545776 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08780904114246368 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4315098226070404 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974514245986938 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001682194648310542 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4811057448387146 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3280148208141327 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4730484187602997 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.995215117931366 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 trainer/global_step‚4835 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3790196478366852 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901949167251587 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12567712366580963 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5090395212173462 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8321427702903748 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21063217520713806 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.33190345764160156 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07650607079267502 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019471870735287666 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20851001143455505 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.056862663477659225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9687979817390442 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.094744473695755 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3600880205631256 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15276078879833221 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.34064289927482605 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828590750694275 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3284652531147003 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31704655289649963 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.37644705176353455± * ð^îò¤ A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.05468036234378815 jet/accuracy_5_of_5‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5191912651062012 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3824882507324219 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005376344081014395 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3914516866207123 -"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.349086731672287 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16872581839561462 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41837137937545776 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08780904114246368 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4315098226070404 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974514245986938 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001682194648310542 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4811057448387146 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3280148208141327 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4730484187602997 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.995215117931366 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 trainer/global_step‚4835 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3790196478366852 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9901949167251587 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12567712366580963 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5090395212173462 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8321427702903748 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21063217520713806 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.33190345764160156 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07650607079267502 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019471870735287666 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20851001143455505 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.056862663477659225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9687979817390442 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.094744473695755 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_5_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3600880205631256 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15276078879833221 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.34064289927482605 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828590750694275 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3284652531147003 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31704655289649963 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.37644705176353455 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4255070388317108 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1524774730205536 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49904587864875793 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036677729338407516 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0647888034582138 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3334835171699524 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923790097236633 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9923111200332642 +jet/accuracy_2_of_2‚0.08743182569742203 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.028396407142281532 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47261396050453186 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42080479860305786 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 particle/accuracy_5_of_6‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.034843139350414276 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08054172247648239 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03275733441114426 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12064436823129654 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_1‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35090020298957825 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11386997252702713 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36669203639030457 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17146049439907074 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.33974358439445496 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9822238683700562 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2875676453113556 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 epoch‚3 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3973034620285034 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_2_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 particle/accuracy_2_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4393034279346466 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44333332777023315 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910547137260437 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09567586332559586 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11314688622951508 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_3_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.898796796798706 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28815510869026184 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08891750127077103 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011791383847594261 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3402356803417206 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_5‚0.4317525029182434 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 +jet/accuracy_2_of_4‚0.27981382608413696 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17475007474422455 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4413047730922699 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.39099201560020447 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_2‚0.6714832186698914 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4116441309452057 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_6‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9492486119270325 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15580807626247406 particle/accuracy_3_of_7‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963880777359009 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31206944584846497 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2611345052719116 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6305399537086487 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3573593199253082 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818966031074524 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30614033341407776 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.37731480598449707 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3045019209384918 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09695566445589066 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33528760075569153 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4534491300582886 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.338116854429245 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.288170725107193 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12660819292068481 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9887623190879822 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2928254306316376 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14774072170257568 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3333333134651184 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3322916626930237 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_3‚0.19288690388202667 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4881410300731659 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881099462509155 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2101198136806488 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0037830278743058443 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09405897557735443 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.46339288353919983 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 particle/accuracy_5_of_7‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4262690246105194 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666913032532 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887279868125916 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0090773431584239 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09465614706277847 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0730452686548233 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0900377407670021 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14460375905036926 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19443435966968536 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12361437827348709 7 Purity/Omp‰31FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12001486122608185 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39756590127944946 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_4_of_7‚0 _step‚395 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.436507910490036 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2947089672088623 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5301971435546875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07650607079267502 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.006194940768182278 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15051838755607605 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *jet/accuracy_1_of_6‚0.9615384340286255 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.412241667509079 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010107816196978092 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26875001192092896 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7742425203323364 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3849570155143738 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35956788063049316 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10481394827365875 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1361110955476761 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39032116532325745 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 'particle/f_score‚0.3435252904891968 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3588513433933258 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014545678161084652 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3793613314628601 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39990660548210144 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.163059264421463 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚50.22644531069847 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3322724401950836 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.46694645285606384 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4694444239139557 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_1‚0.4181266725063324 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899292588233948 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979872107505798 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013287810608744621 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3057556748390198 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13879527151584625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3864942491054535 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07333333045244217 particle/accuracy_6_of_6‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36116939783096313 particle/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02767537347972393 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5253136157989502 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9862969517707825 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26969268918037415 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20896799862384796 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15859584510326385 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.31308698654174805 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3336600959300995 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2244795858860016 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39058879017829895 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24537040293216705 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896868467330933 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4574255347251892 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6655310988426208 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.818370521068573 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14396552741527557 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0891065001487732 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884447715098064e+09 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3167289197444916 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34162840247154236 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39990660548210144 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02440476231276989 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9837095737457275 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14760740101337433 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07375479489564896 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15054979920387268 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06895040720701218 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09705746918916702 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07112908363342285 ,particle/sensitivity‚0.26437267661094666 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10021157562732697 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3662037253379822 jet/accuracy_3_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3686080873012543 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 'particle/accuracy‚0.842323899269104 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14451861381530762 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.32364338636398315 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012599708512425423 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.27812495827674866 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13139815628528595 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3360566198825836 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 -jet/accuracy_4_of_4‚0.0057404013350605965 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 _runtime‚4488.560030004 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_5_of_6‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09668289870023727 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3579123318195343‹ψi°“ï‚(:ˆ î€æÔ›ùÕÇ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.909744262695312* gpu.0.memoryAllocatedBytes‚ 12807372800 gpu.0.temp‚37.0 gpu.0.powerWatts‚190.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.1238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…5Pð‚(:Å î€æÔÉÎÇÊ disk.drbd2266.in‚0# proc.memory.rssMB‚ 2727.82421875 cpu‚25.310911595484413 network.sent‚ 1.1865697e+07 disk.dm-1.out‚ 170.890625 disk.dm-0.in‚0.078125 disk.sda1.out‚0) disk./.usagePercent‚19.86989702230059$ memory_percent‚89.62116241455078 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚296.0758247375488 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚19074.5087890625 disk.drbd2080.in‚0 disk.rbd1.out‚ 31.38671875 disk.rbd0.in‚0* proc.memory.percent‚16.649317741394043 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.sda.out‚ 170.890625 disk.drbd2265.out‚0 network.recv‚ 1.626449e+06' disk.nvme0c0n1.out‚19074.5087890625 disk.sda1.in‚0& proc.memory.availableMB‚ 1700.46875 disk.sda3.out‚ 170.890625 disk.nvme0n1.in‚0.078125" disk.dm-0.out‚19074.5087890625_5ô2ñ- _wandbruntime‚4505  _runtime‚4505ÎúÈò ' lr-AdamW/pg2‚0.00011527946834411901 trainer/global_step‚4849 % _timestamp‚1.7884447881423845e+09 _runtime‚ 4505.18969997 _step‚396 ' lr-AdamW/pg1‚0.00011527946834411901dˆ–¦ÍóÇ ' lr-AdamW/pg1‚0.00011527946834411901 ' lr-AdamW/pg2‚0.00011527946834411901 trainer/global_step‚4849 % _timestamp‚1.7884447881423845e+09 _runtime‚ 4505.18969997 _step‚396Œìº+2ô- _wandbruntime‚4505  _runtime‚45050Ä:§ö'õðO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348851999464960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339443181198960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344253758635520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _runtime‚4505.190552068 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191712482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4838636574208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334862532852736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348851199169024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4579850106100 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4580631306864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348851999464960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105778692 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105755582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85956900 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344253758635520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50575024JFÈAföõO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 _runtime‚4505.190552068 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191712482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4838636574208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334862532852736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348851199169024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4579850106100 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4580631306864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17170432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348851999464960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105778692 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105755582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85956900 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344253758635520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50575024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1031 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348851999464960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371895808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339443181198960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344253758635520ÇǃW2÷- _wandbruntime‚4506  _runtime‚4506·‡Ð'øúM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85973940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191750856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334929367681024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4598368018944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383613206329344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4839803642880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884447889588895e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85973942 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348921558537216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4581555263476 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105800052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344322390209024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85973940 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76167969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344322407379456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344322407379456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85973942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191773994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105776916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378773377913856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191773994 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344322390209024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4599151157760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582953 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339510922944500 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4598368018944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25583449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378773402686464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ãšLÆ'ùÿM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334929350641664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50585019 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4580774062504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191750856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334929367681024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4598368018944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383613206329344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4839803642880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884447889588895e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85973942 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348921558537216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4581555263476 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105800052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344322390209024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85973940 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76167969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344322407379456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344322407379456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85973942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191773994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105776916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378773377913856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 800309248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191773994 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344322390209024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189939712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4599151157760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582953 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339510922944500 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4598368018944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25583449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378773402686464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85973940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86808902 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.–‰S˜ñ all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338203422299136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚387369393641984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106848744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4626061517508 trainer/global_step‚4899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51074627 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347685337310720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106848744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347685354481152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4887563629056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352329166530048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352329166530048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4643829219328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4548.330677894 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51074624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208434176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193634534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4887379967488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884448312831907e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352329966825984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚387369185207808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338203439338496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352329966825984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4644612344832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4643829219328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183661568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382481805240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264±?èû'ˆõO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4644612344832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4643829219328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183661568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 798240124 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 781200764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11234443264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382481805240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 172325376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11234443264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382481830012928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106825634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12013121024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347685337310720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86808900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25827244 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342829483816644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 800295936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4644612344832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1010 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4626842718272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86808900 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76902397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86808902 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338203422299136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚387369393641984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106848744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 783125504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 783125504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12259950592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1010 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4626061517508 trainer/global_step‚4899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51074627 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23110 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3157 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347685337310720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106848744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12259950592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347685354481152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3157 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4887563629056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352329166530048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352329166530048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4643829219328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10672704512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11048832512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4548.330677894 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1019225937 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11234443264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51074624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208434176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1021188096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11635030865 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12013121024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193634534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24772608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4887379967488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1010 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1010 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚521 % _timestamp‚1.7884448312831907e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352329966825984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚387369185207808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338203439338496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352329966825984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 800295936‘zÕ2‰- _wandbruntime‚4549  _runtime‚4549¬–ÿK'ŠúM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76917223 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347753968884224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192036864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342897225561976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387445838371840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884448320900087e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387446055181312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644756704768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352399525898240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338270257127424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4645539843584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106846968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86825940 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4888542978048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 _step‚402 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342898023802308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 783138816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4645539843584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193696046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51084619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25832076 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382557487430656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4549.137874359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352398725588992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347753986054656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1051 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1019225937 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352399525898240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11048832512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338270240088064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4627766674884 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193672908 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51084616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4626985473912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1010 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12013121024 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106846968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1021188096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76916692 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86825942 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11635030865 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382557512203264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 798240332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193672908 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193696046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371895808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24772608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25832604 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216809472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1010 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86825942 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347753986054656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17170432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86825940 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347753968884224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157¦nÑá¿‹ÿM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347753968884224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76917223 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11234443264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347753968884224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192036864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342897225561976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1031 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387445838371840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884448320900087e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12013121024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387446055181312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12259950592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12259950592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644756704768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352399525898240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11234443264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1021188096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1010 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10672704512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12259950592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1010 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 783138816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338270257127424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 800309248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4645539843584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106846968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86825940 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11048832512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4888542978048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302690304 _step‚402 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342898023802308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11234443264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17170432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚521 SEDeviceStatsMonitor.on_train_batch_en