:W&B E?M r6jnrpi2; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0'=" 0k2f2q1yw95i r6jnrpi2 r6jnrpi2juju11-uc-san-diegoSPAtop" _wandb{}  batch_size512  dataset_limit0.2  dropout0.26318099207389106 epochs10  hidden_dim256 ' learning_rate0.0011777537344912284  num_attention_heads4  num_branch_encoder_layers3  num_embedding_layers6  num_encoder_layers2Bjjtttt_blocks_r6jnrpi2bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5p ՟3 (Hhp"3.12.10*0.29.0j linux-x86_64ҲS7 r6jnrpi2&  _runtime0  _wandb.runtime0XgRj -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 ԟ埘*-of*=/data/spatop/options_files/sweep_tttt_jj/config_r6jnrpi2.json*-l* /scratch/logs*-n*jjtttt_blocks_r6jnrpi22-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/python i6rat52puk4a2i5vupd9tkbpj4pb1brx=NVIDIA A100-SXM4-80GB /.#ӽS NVIDIA A100-SXM4-80GB6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c313.2 i6rat52puk4a2i5vupd9tkbpj4pb1brx2  requirements.txt=W r6jnrpi2ZF (H (H (hp"3.12.10*0.29.0j linux-x86_64!P=@Y r6jnrpi2ZH (H (H (8hp"3.12.10*0.29.0j linux-x86_64e&* r6jnrpi2b trainer/global_step2(hX\ r6jnrpi2ZK (H (H(8hp"3.12.10*0.29.0j linux-x86_64 bt/ r6jnrpi2b*"trainer/global_step2ԁũJ r6jnrpi2j9 ֟**LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] I~ r6jnrpi2j  ֟ȇ ┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 2.6 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 151 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ȭ"\ r6jnrpi2j ֟χTrainable params: 157 M Non-trainable params: 86 Total params: 157 M Total estimated model params size (MB): 630.532 Modules in train mode: 4316 Modules in eval mode: 0 Total FLOPs: 0 Xhq r6jnrpi2j` ֟ܯPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). w)w{ r6jnrpi2jj ֟ęZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). S;y r6jnrpi2jh ͙֟Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). 4 r6jnrpi2jo ֟ՙ_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). pz r6jnrpi2ji ֟ݙYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). Y)`t r6jnrpi2jc ֟Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). Lq r6jnrpi2j` ֟Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). "Swt r6jnrpi2jc ֟ȼSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ̆n r6jnrpi2j] ֟Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). m r6jnrpi2j\ ֟Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). `} r6jnrpi2*  transformer_dim256  transformer_dim_scale2.0  initial_embedding_dim256  position_embedding_dim256 ! num_branch_embedding_layers3  num_jet_embedding_layers0  num_jet_encoder_layers2  num_detector_layers2  num_regression_layers3  num_classification_layers3  split_symmetric_attention1 ! transformer_activation"gelu"  skip_connections1 ( "initial_embedding_skip_connections1  linear_block_type"GRU"  transformer_type"Gated"  linear_activation"gelu"  normalization "LayerNorm"  masking "Filling" linear_prelu_activationtrue  use_pairwise_interactions1  num_pairwise_features4  pairwise_embedding_dim8  pairwise_input_source"all"  pairwise_block_embeddings1  pairwise_cross_typetrue  pairwise_feature_set""  pairwise_eq2to2false  use_moefalse  num_experts8  num_experts_per_tok2  moe_loss_scale0.01 F event_info_file2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file""  testing_file""  normalize_features1  limit_to_num_jets0  balance_particles0  balance_jets0 ! balance_classificationsfalse  partial_events1  dataset_randomization0  train_validation_split0.95  limit_val_batches1.0  num_dataloader_workers0  mask_sequence_vectors1  combine_pair_loss"min"  optimizer"AdamW"  focal_gamma0.0  combinatorial_scale0.0 # learning_rate_warmup_epochs1.0  learning_rate_cycles1  assignment_loss_scale1.0  detection_loss_scale1.0  kl_loss_scale0.0  regression_loss_scale0.0 ! classification_loss_scale0.0 ! use_mass_classificationfalse  mass_classes[]  balance_lossestrue  l2_penalty0.0002  gradient_clip10.0 num_gpu1  verbose_outputfalse  usable_gpus""  trial_time"" $ trial_output_dir"./test_output"Y织 r6jnrpi2j ֟/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. N r6jnrpi2j ֟/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ϥ r6jnrpi2j ۟מ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. DP r6jnrpi2j ۟/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. y9(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.49275207519531* gpu.0.memoryAllocatedBytes 40795963392 gpu.0.temp40.0 gpu.0.powerWatts230.819' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.1638 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0fA(:  network.recv41590) disk./.usagePercent76.85252120725453 disk.sda3.in0 disk.sda2.out0 disk.nvme0c0n1.in0 disk.dm-1.in0 disk.dm-1.out 0.4921875 disk.nvme0n1.in0$ disk./.usageGB1145.158103942871 disk.sda.out 0.4921875 disk.sda2.in0 disk.drbd1260.in0 disk.drbd1260.out 0.00390625$ disk.nvme0n1.out312.78759765625 disk.dm-0.in0! disk.dm-0.out312.78759765625 network.sent33702 disk.sda1.in0 disk.sda1.out0$ memory_percent72.39785194396973( proc.memory.availableMB 4522.3359375# proc.memory.rssMB 3019.62109375* proc.memory.percent18.430304527282715 cpu12.345205938604613 disk.sda3.out 0.4921875 disk.sda.in0& disk.nvme0c0n1.out312.78759765625 proc.cpu.threads17<^ (: ( proc.memory.availableMB 4520.7578125 cpu13.880401655179318 proc.cpu.threads17 disk.dm-1.out 0.8671875 disk.dm-0.in 90.27734375! disk.dm-0.out505.33447265625 disk.sda1.in0 disk.sda3.in0 disk.sda.out 0.8671875# proc.memory.rssMB 3019.70703125) proc.memory.percent18.43082904815674 network.sent44061$ disk./.usageGB1145.658046722412 disk.sda2.in0& disk.nvme0c0n1.out505.33447265625 disk.drbd1260.in0$ disk.nvme0n1.out505.33447265625 disk.sda1.out0 disk.sda3.out 0.8671875 network.recv46528) disk./.usagePercent76.88607278666946 disk.sda2.out0! disk.nvme0c0n1.in 90.27734375 disk.dm-1.in0 disk.drbd1260.out 0.00390625 disk.sda.in0$ memory_percent72.40748405456543 disk.nvme0n1.in 90.27734375!(:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.50251770019531* gpu.0.memoryAllocatedBytes 40804352000 gpu.0.temp42.0 gpu.0.powerWatts230.565' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.113 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0wկҒ"(:  gpu.0.gpu72.0 gpu.0.memory13+ gpu.0.memoryAllocated47.50984191894531* gpu.0.memoryAllocatedBytes 40810643456 gpu.0.temp44.0 gpu.0.powerWatts238.082' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.6164 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0d+\#(:  network.sent49642 network.recv50826) disk./.usagePercent76.87771772478199% disk./.usageGB1145.5335502624512 disk.drbd1260.out 0.0078125 disk.sda3.out 1.35546875) proc.memory.percent18.43109130859375 proc.cpu.threads17& disk.nvme0c0n1.out674.54345703125 disk.nvme0n1.in 125.2578125$ disk.nvme0n1.out674.54345703125 disk.sda3.in0 disk.sda.in0 disk.sda2.out0 cpu15.147929874076551! disk.nvme0c0n1.in 125.2578125 disk.dm-1.in0 disk.dm-1.out 1.35546875 disk.drbd1260.in0! disk.dm-0.out674.54345703125 disk.sda.out 1.35546875) proc.memory.availableMB 4520.37890625 proc.memory.rssMB3019.75 disk.dm-0.in 125.2578125 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0$ memory_percent72.40979671478271\$(: Ţ gpu.0.gpu93.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51228332519531* gpu.0.memoryAllocatedBytes 40812740608 gpu.0.temp44.0 gpu.0.powerWatts245.828' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.1656 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0fk[%(:  disk.dm-0.in 125.2578125! disk.dm-0.out770.90283203125 disk.sda1.in0 disk.sda3.out 1.77734375 disk.sda.in0 disk.sda.out 1.77734375$ disk./.usageGB1145.556999206543 disk.drbd1260.in0 disk.sda3.in0& disk.nvme0c0n1.out770.90283203125# memory_percent72.4233627319336& proc.memory.availableMB 4518.15625 cpu16.15294018965366) disk./.usagePercent76.87929140309497 disk.drbd1260.out 0.0078125 disk.nvme0n1.in 125.2578125 disk.sda2.out0" proc.memory.rssMB 3019.7578125 network.sent69068 disk.sda1.out0 disk.sda2.in0! disk.nvme0c0n1.in 125.2578125 disk.dm-1.in0 disk.dm-1.out 1.77734375) proc.memory.percent18.43113899230957 proc.cpu.threads17 network.recv80774$ disk.nvme0n1.out770.902832031256đ&(: ڶ gpu.0.gpu26.0 gpu.0.memory6+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts320.591' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.1182 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0>ɾrF'(: ˧ disk.sda.out2.25! proc.memory.rssMB 3019.765625) disk./.usagePercent76.87930010735113 disk.nvme0n1.in 125.2578125 disk.dm-0.out 1276.64453125 disk.sda1.in0 disk.sda2.out0 proc.cpu.threads17 network.recv85635 disk.dm-1.in0 disk.sda1.out0 disk.sda3.out2.25 disk.sda.in0) proc.memory.percent18.43118667602539 network.sent75463# disk./.usageGB1145.55712890625 disk.dm-1.out2.25" disk.nvme0n1.out 1276.64453125 disk.sda2.in0$ memory_percent72.41904735565186) proc.memory.availableMB 4518.86328125 cpu16.98768825784232! disk.nvme0c0n1.in 125.2578125$ disk.nvme0c0n1.out 1276.64453125 disk.drbd1260.in0 disk.drbd1260.out 0.01171875 disk.dm-0.in 125.2578125 disk.sda3.in0:yH3(b/ lr-AdamW/pg1"trainer/global_step2:JP>Ρ3)b/ lr-AdamW/pg2"trainer/global_step2:JP?3-*) _wandbruntime89  _runtime89͓+ & lr-AdamW/pg29.554624667230164e-05 trainer/global_step49 % _timestamp1.7884651993199632e+09 _runtime 89.360658743 _step0 & lr-AdamW/pg19.554624667230164e-05g, & lr-AdamW/pg29.554624667230164e-05 trainer/global_step49 % _timestamp1.7884651993199632e+09 _runtime 89.360658743 _step0 & lr-AdamW/pg19.554624667230164e-05jh-bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP7SSr.bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP83g/bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP7ig0bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP_f1bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPzSn2bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPU+l3bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPkTf4bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP$Ltl5bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP&Me6ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPR48p7bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPβW8bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP2ee9ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPj~s:bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP/ih;bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP^ibY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP:Ts?bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP@go@bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP iʣ^AbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPYknBbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPH=nCbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPmgDbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPqEbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPuyhFbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPYO?iGbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPspjHbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP\kIbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP[VVtgJbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPh]nKbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPnhrLbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPhMbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPz-AvNbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPR{KiObe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPhnPbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP)KmQbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP6gRbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPnSbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP%nTbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPMnUbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPKoVbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP}uHnWbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP.&$fXbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPYnYbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPoZbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPp[bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP#rh\bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP8[ c]b_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPAmdbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPCwpebl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP-bfb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPw4rgbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP gmbhb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP鿻mibi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPx;4sjbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPt[mkbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP#?ilbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPlmbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP inbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPqobm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP&ppbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP aqb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP:.Xgrbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP]dsb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPHYjtbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPk9cub_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP_bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPY Q`b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPq:kbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP -mbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP4nab\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPuFtfba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPflobj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPbybt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP'wbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPgbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPab\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP>mbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP5eb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPkbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP mbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP$ QDpbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPƒesbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPP`b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP;ʺBibd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPagbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPvpqbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPShbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP4kbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP/1\.) _wandbruntime89  _runtime89  O KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 2422442292736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 % _timestamp1.7884651993216448e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22386 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated10611813169152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed701400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 2422471390720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated10194523832848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed917542 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated10611813169152 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed569722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated1637338 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1181 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed345380 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 trainer/global_step49 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed10611787741184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22386 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed10307097862680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 115113819888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 112599457800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_b̺ytes.all.allocated 2570389977088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _runtime 89.362526895 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated701406 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142089728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated701406 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated345846 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak860 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005133 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142089728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed10194498404880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730044 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed224342 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current86602752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 115265542656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22685 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 115265542656 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed 2570303374336 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated10727078711808 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 112749841920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _step1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed917542 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed1618942 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed10724537583104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 147918586368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current57504768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated10309637652736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated935932 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated10727078711808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed10611787741184 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated224345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed10724537583104 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672d'O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142089728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed10194498404880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730044 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed224342 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current86602752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 115265542656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22685 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 115265542656 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed 2570303374336 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated10727078711808 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 112749841920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _step1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed917542 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed1618942 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed10724537583104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 147918586368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current57504768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated10309637652736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated935932 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated10727078711808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed10611787741184 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated224345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed10724537583104 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 2422442292736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 % _timestamp1.7884651993216448e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22386 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated10611813169152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed701400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 2422471390720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated10194523832848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed917542 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated10611813169152 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed569722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated1637338 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1181 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed345380 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 trainer/global_step49 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed10611787741184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22386 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed10307097862680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 115113819888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 112599457800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated 2570389977088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _runtime 89.362526895 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated701406 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142089728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated701406 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated345846 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak860 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005133 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464q0Z(: ܋ disk.sda2.out0 disk.nvme0n1.in 125.2578125" disk.nvme0n1.out 1670.45703125& proc.memory.availableMB 4518.09375) proc.memory.percent18.43132972717285 network.recv91055 disk.sda1.in0 disk.sda3.in0! disk.nvme0c0n1.in 125.2578125 disk.drbd1260.out 0.01171875 proc.cpu.threads17 network.sent82092% disk./.usageGB1145.7196311950684$ disk.nvme0c0n1.out 1670.45703125$ memory_percent72.42374420166016" proc.memory.rssMB 3019.7890625 cpu17.676636993123683) disk./.usagePercent76.89020577230224 disk.dm-0.out 1670.45703125 disk.sda1.out0 disk.sda3.out 2.98046875 disk.sda.out 2.98046875 disk.dm-1.in0 disk.dm-1.out 2.98046875 disk.drbd1260.in0 disk.dm-0.in 125.2578125 disk.sda.in0 disk.sda2.in0*M(:  gpu.0.gpu71.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts235.916' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.1832 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ubb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPHcb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP>BRobj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPu+nbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP)- mbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP!fba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP)+gbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP:O$qbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP obj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPkbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPueb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP ubp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPHGsbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPU)+ibd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP-E_bZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP3rbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPiRpbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP-t6lbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPRnbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPa'MHjbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPI6mbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP##X`b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPtЭVbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP}jbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP gfba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP2db_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPD eb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPmbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP(Zobj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP|rbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPKobj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP:qbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPI3^bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPC*db_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP5\bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPeb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPtkbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP]bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPY^Wibd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPҏgbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP5ubp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPqhbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPlkbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPgP;]bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP+kgbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPLkrbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPvjbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPgNobj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP!-db_ obj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPQ{_`b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP+ibd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP9ibd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPFlbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPu(cb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP ާibd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP)mbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP) mbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP qbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPŢhbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP,0eb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPXxmbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP|0Mkbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP\(fba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP]ulbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPsbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPKjbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP'X^fba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPR|rbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP_pbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP&dMmbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP^bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP\!gfba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPB hbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPUpbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP~nbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPKrbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP mbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP*kbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPPDOkbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPtT gbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPG{jbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPfvom.) _wandbruntime90  _runtime90>x2M LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated954309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current57491456 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated10940344316928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed10822778116608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 117386074950 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253c%##952 _step2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated352665 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed715281 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated715287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed580882 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 150783855104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated10397052542432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed10822778116608 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 117540772352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 trainer/global_step49 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated10940344316928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated1669596 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated10822803544576 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 115025058304 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated954309 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed10937803174912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated228677 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 117540772352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 2466503662080 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak860 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed935893 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed228674 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated10822803544576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed1651174 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated1669596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 115025058304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687?B&M >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed935893 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed228674 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated10822803544576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed1651174 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated1669596 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 115025058304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated954309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current57491456 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated10940344316928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed10822778116608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 117386074950 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 _step2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated352665 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed715281 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated715287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed580882 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 150783855104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated10397052542432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed10822778116608 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 117540772352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 trainer/global_step49 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated10940344316928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated1669596 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated10822803544576 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 115025058304 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated954309 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed10937803174912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated228677 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 117540772352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 2466503662080 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak860 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968krt@b; loss/FRt1/detection_loss"trainer/global_step2:JPm .@b; loss/FBt3/detection_loss"trainer/global_step2:JPPAb< loss/FBt2/assignment_loss"trainer/global_step2:JP.x@b; loss/FRt3/detection_loss"trainer/global_step2:JPCBvBb= loss/SRqqt3/detection_loss"trainer/global_step2:JP\zrBb= loss/SRqqt2/detection_loss"trainer/global_step2:JPgBb= loss/SRqqt4/detection_loss"trainer/global_step2:JPd(7b2 loss/total_loss"trainer/global_step2:JPPAb< loss/FRt1/assignment_loss"trainer/global_step2:JPZcMAb< loss/FRt3/assignment_loss"trainer/global_step2:JP׸pCb> loss/SRqqt1/assignment_loss"trainer/global_step2:JP+Ab< loss/FBt3/assignment_loss"trainer/global_step2:JPJװCb> loss/SRqqt3/assignment_loss"trainer/global_step2:JP loss/SRqqt4/assignment_loss"trainer/global_step2:JP4Y@b; loss/FBt2/detection_loss"trainer/global_step2:JPAbMAb< loss/FBt1/assignment_loss"trainer/global_step2:JP *x@b; loss/FRt4/detection_loss"trainer/global_step2:JPbCb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPo.) _wandbruntime90  _runtime90UlK $ _timestamp1.788465200857599e+09 /loss/FBt4/detection_loss1.6332666750763598 epoch0 /loss/FBt4/assignment_loss0.585139579174662 .loss/FRt2/detection_loss5.182186569486346 0loss/FRt2/assignment_loss7.0414988824299405 /loss/FRt4/assignment_loss7.362927694881663 0loss/SRqqt1/detection_loss2.879182266608133 _step3 1loss/SRqqt4/assignment_loss5.181996674373232 /loss/FBt2/detection_loss12.111624448463834 /loss/FBt1/assignment_loss0.873952372206582 .loss/FRt4/detection_loss4.147311646973385 1loss/SRqqt2/assignment_loss5.248566342724694 /loss/FRt1/detection_loss2.6671779140668073 .loss/FBt3/detection_loss9.832512228853172 0loss/FBt2/assignment_loss0.9097086961927086 /loss/FRt3/detection_loss3.1448921808884256 0loss/SRqqt3/detection_loss2.067855137988185 _runtime 90.901380843 1loss/SRqqt2/detection_loss4.7583993346326885 0loss/SRqqt4/detection_loss3.823627380804084 &loss/total_loss118.17110353674676 0loss/FRt1/assignment_loss7.1409974429918375 /loss/FRt3/assignment_loss7.452672100067138 1loss/SRqqt1/assignment_loss5.089940224663686 0loss/FBt3/assignment_loss0.6259599779215124 trainer/global_step49 1loss/SRqqt3/assignment_loss5.276959283977535 /loss/FBt1/detection_loss13.132748481300142JO /loss/FBt1/assignment_loss0.873952372206582 .loss/FRt4/detection_loss4.147311646973385 1loss/SRqqt2/assignment_loss5.248566342724694 /loss/FRt1/detection_loss2.6671779140668073 .loss/FBt3/detection_loss9.832512228853172 0loss/FBt2/assignment_loss0.9097086961927086 /loss/FRt3/detection_loss3.1448921808884256 0loss/SRqqt3/detection_loss2.067855137988185 _runtime 90.901380843 1loss/SRqqt2/detection_loss4.7583993346326885 0loss/SRqqt4/detection_loss3.823627380804084 &loss/total_loss118.17110353674676 0loss/FRt1/assignment_loss7.1409974429918375 /loss/FRt3/assignment_loss7.452672100067138 1loss/SRqqt1/assignment_loss5.089940224663686 0loss/FBt3/assignment_loss0.6259599779215124 trainer/global_step49 1loss/SRqqt3/assignment_loss5.276959283977535 /loss/FBt1/detection_loss13.132748481300142 $ _timestamp1.788465200857599e+09 /loss/FBt4/detection_loss1.6332666750763598 epoch0 /loss/FBt4/assignment_loss0.585139579174662 .loss/FRt2/detection_loss5.182186569486346 0loss/FRt2/assignment_loss7.0414988824299405 /loss/FRt4/assignment_loss7.362927694881663 0loss/SRqqt1/detection_loss2.879182266608133 _step3 1loss/SRqqt4/assignment_loss5.181996674373232 /loss/FBt2/detection_loss12.111624448463834#N(: ( disk./.usagePercent76.8339972700995 disk.sda3.out3.5625 disk.sda.out3.5625 disk.nvme0c0n1.in 274.46875 disk.dm-0.in 274.46875$ memory_percent72.42178916931152( proc.memory.availableMB 4518.4140625* proc.memory.percent18.431377410888672 network.recv97851 disk.sda3.in0& disk.nvme0c0n1.out2091.9462890625 disk.dm-1.in0 disk.drbd1260.in0 disk.nvme0n1.in 274.46875 disk.sda1.out0! proc.memory.rssMB 3019.796875% disk./.usageGB1144.8820838928223 disk.sda2.out0$ disk.nvme0n1.out2091.9462890625 disk.sda1.in0 network.sent115016 disk.sda.in0 disk.sda2.in0 disk.dm-1.out3.5625 disk.drbd1260.out0.015625! disk.dm-0.out2091.9462890625 cpu18.270271960342853 proc.cpu.threads17oh(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp42.0 gpu.0.powerWatts116.629' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.3258 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0'(: ͠ gpu.0.gpu73.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts224.162' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent44.8324 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0IlL(: ͠ disk.dm-0.in 274.46875 disk.sda1.in0 disk.sda1.out0 disk.sda.out4.375$ disk./.usageGB1144.886875152588$ disk.nvme0n1.out2116.2470703125 disk.sda3.in0 disk.sda.in0 disk.sda2.out0 disk.nvme0c0n1.in 274.46875 disk.dm-1.in0$ memory_percent72.42343425750732 disk.sda3.out4.375 disk.sda2.in0 disk.dm-1.out4.375 disk.drbd1260.in0 disk.drbd1260.out0.015625! proc.memory.rssMB 3019.796875 proc.cpu.threads17! disk.dm-0.out2116.2470703125& disk.nvme0c0n1.out2116.2470703125) proc.memory.availableMB 4518.14453125* proc.memory.percent18.431377410888672 cpu18.77155700438553 network.sent154532 network.recv103947) disk./.usagePercent76.83431881556251 disk.nvme0n1.in 274.46875(: ܠ gpu.0.gpu64.0 gpu.0.memory14+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts249.288' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.857600000000005 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0jb(: ܠ disk.sda2.in0 disk.drbd1260.in0# memory_percent72.4236011505127# proc.memory.rssMB 3019.80078125) disk./.usagePercent76.83452822972552 disk.nvme0n1.in 274.46875 disk.sda1.out0 disk.dm-1.out 4.80078125 disk.drbd1260.out 0.01953125* proc.memory.percent18.431401252746582 cpu19.212647834442546 proc.cpu.threads17 network.sent160529% disk.nvme0n1.out2383.18115234375 disk.sda.out 4.80078125( proc.memory.availableMB 4518.1171875 disk.dm-0.in 274.46875" disk.dm-0.out2383.18115234375 disk.sda1.in0 disk.sda3.in0 disk.sda2.out0 disk.nvme0c0n1.in 274.46875' disk.nvme0c0n1.out2383.18115234375 disk.dm-1.in0 network.recv108337% disk./.usageGB1144.8899955749512 disk.sda3.out 4.80078125 disk.sda.in0٧(:  gpu.0.gpu79.0 gpu.0.memory14+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts242.365' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.473 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors09?b(:  disk.sda1.in0 disk.drbd1260.out 0.01953125 network.sent167029 disk.sda2.out0( proc.memory.availableMB 4517.8515625# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582% disk./.usageGB1144.9526824951172 disk.sda1.out0 disk.sda3.out 5.16015625 disk.sda.in0 disk.sda.out 5.16015625 disk.drbd1260.in0$ memory_percent72.42522239685059 cpu19.59663580986335 disk.sda3.in0 disk.sda2.in0 disk.nvme0c0n1.in 274.46875' disk.nvme0c0n1.out2622.72802734375 disk.dm-1.in0 disk.dm-1.out 5.16015625 proc.cpu.threads17 network.recv113252) disk./.usagePercent76.83873520153568 disk.nvme0n1.in 274.46875% disk.nvme0n1.out2622.72802734375 disk.dm-0.in 274.46875" disk.dm-0.out2622.72802734375E](:  disk.dm-1.in0" disk.dm-0.out3066.93310546875 disk.sda1.out0* proc.memory.percent18.431401252746582 disk.sda3.out 5.8046875 disk.sda.in0' disk.nvme0c0n1.out3066.93310546875 disk.drbd1260.in0% disk.nvme0n1.out3066.93310546875# proc.memory.rssMB 3019.80078125 cpu19.941035548473167 proc.cpu.threads17 network.sent172500% disk./.usageGB1144.9523658752441 disk.sda2.out0 disk.dm-1.out 5.8046875 disk.dm-0.in411.8125) proc.memory.availableMB 4518.23828125 network.recv117121) disk./.usagePercent76.83871395291034 disk.sda3.in0 disk.drbd1260.out 0.0234375 disk.nvme0n1.in411.8125 disk.sda1.in0$ memory_percent72.42286205291748 disk.sda.out 5.8046875 disk.sda2.in0 disk.nvme0c0n1.in411.8125Mȓ(:  gpu.0.gpu37.0 gpu.0.memory10+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts329.179' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.8358 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Ӓ_0+ _wandbruntime173  _runtime173ʂ _step4 ' lr-AdamW/pg10.00019304241674607882 ' lr-AdamW/pg20.00019304241674607882 trainer/global_step99 % _timestamp1.7884652831076264e+09 _runtime 173.148222606X[k _step4 ' lr-AdamW/pg10.00019304241674607882 ' lr-AdamW/pg20.00019304241674607882 trainer/global_step99 % _timestamp1.7884652831076264e+09 _runtime 173.148222606匚0+ _wandbruntime173  _runtime173&O HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed1395450 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated1856834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 4625539857920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed20547173256588 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 _runtime 173.149746818 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed21387845001728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed21161306512384 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated686345 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 A8DeviceStatsMonitor.on_train_batch_start/num_device_allockF/3672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 229054190080 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 228753734596 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed21387845001728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current55407616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated20320959312048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed3233894 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current445 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed1838444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated440945 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed1395450 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed685900 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 294227049472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed20320933884080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated21390386130432 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed1126842 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak860 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 294171641856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed21161306512384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed 4919682401792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 _step5 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 229054190080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated1395456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 4625510759936 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated20549713046644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated1127290 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current84505600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 _step5 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 229054190080 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated1395456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 4625510759936 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated20549713046644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated1127290 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current84505600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed1395450 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated1856834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 4625539857920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed20547173256588 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 _runtime 173.149746818 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed21387845001728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed21161306512384 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated686345 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 229054190080 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 228753734596 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed21387845001728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current55407616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated20320959312048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed3233894 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current445 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed1838444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated440945 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed1395450 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed6859000+ _wandbruntime174  _runtime174&M ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated3284548 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed21372296887808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 trainer/global_step99 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed3266126 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated20754514011290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated1409337 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated 4669601227264 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _step6 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated21603651735552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated21372322315776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 % _timestamp1.7884652846672337e+09 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 228813705728 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current84492288 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 297159562240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 231329419776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated1875211 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed692661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 4669572129280 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed1409331 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed1856795 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed3266126 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed 4966676297216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed20523462593664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 4966760789504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated20523488021632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 228813705728 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated1138390 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated3284548 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed1137935 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 297104167936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated1409337 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed445274 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed21601110593536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 231329419776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672! lM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed1409331 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 228813705728 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated1138390 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated3284548 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed1137935 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 297104167936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated1409337 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed445274 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed21601110593536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 231329419776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated3284548 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed21372296887808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 trainer/global_step99 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed3266126 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated20754514011290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated1409337 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated 4669601227264 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _step6 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated21603651735552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated21372322315776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 % _timestamp1.7884652846672337e+09 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 228813705728 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current84492288 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 297159562240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 231329419776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated1875211 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed692661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 4669572129280 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed1409331 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed1856795 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed3266126 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed 4966676297216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed20523462593664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 4966760789504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated20523488021632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 440685683712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated31710850711552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed31710825283584 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated4867242 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated2089506 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current469 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed4848846 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed2089500 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated30789788440552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 trainer/global_step149 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 340327136768 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed32051152420352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 6828579227136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated32053693549056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated32053693549056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 342842837504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 440628178944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _runtime 255.660070792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak860 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 339879287216 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated2777736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed1026856 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed30787248650496 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 340327136768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated 7269294008832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _step9 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated657545S+O HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated32053693549056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 342842837504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 440628178944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _runtime 255.660070792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak860 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 339879287216 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated2777736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed1026856 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed30787248650496 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 340327136768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated 7269294008832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _step9 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated657545 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated1684867 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed1684398 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocatᨀed 440685683712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated31710850711552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed31710825283584 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated4867242 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated2089506 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current469 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed4848846 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed2089500 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated30789788440552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 trainer/global_step149 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 340327136768 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed32051152420352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed 6828579227136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated32053693549056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 0v0+ _wandbruntime257  _runtime257rc+&M I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed2777697 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 345118067200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated 6872669694464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 % _timestamp1.7884653672203033e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 443601022464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed30992049614934 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated4899500 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated2796113 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated2796113 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated1695943 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated31921841086976 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated4899500 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 7316270716928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 342602353152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed4881078 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed1033600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed2777697 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1034066 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 443543531008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated30649923500832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 342151542070 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed661874 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed1695474 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed2103381 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated2103387 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 6872640596480 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 344665904366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step10 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed31921815659008 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated32266959154176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 trainer/global_step149 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated32266959154176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1181 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed 7316184127488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated30994589405198 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed4881078 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 345118067200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak8607&M XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step10 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed31921815659008 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated32266959154176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 trainer/global_step149 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated32266959154176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1181 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed 7316184127488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated30994589405198 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed4881078 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 345118067200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak860 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed2777697 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 345118067200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated 6872669694464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 % _timestamp1.7884653672203033e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 443601022464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed30992049614934 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated4899500 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated2796113 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated2796113 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated1695943 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated31921841086976 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated4899500 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 7316270716928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 342602353152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed4881078 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed1033600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed2777697 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1034066 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 443543531008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated30649923500832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 342151542070 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed661874 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed1695474 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed2103381 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated2103387 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed 6872640596480 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 344665904366 k0+ _wandbruntime257  _runtime257r?O 1loss/FBt2/assignment_loss0.49779384104695057 .loss/FRt2/detection_loss3.599359381583429 1loss/SRqqt4/detection_loss2.1840746219215146 0loss/SRqqt1/detection_loss2.198344301711487 /loss/FBt1/detection_loss2.8172087023163788 /loss/FBt4/detection_loss7.2707676630450635 %loss/total_loss93.58928491635456 _runtime 257.262828404 1loss/SRqqt3/assignment_loss3.867203722397486 0loss/FBt1/assignment_loss0.3336396885817953 .loss/FBt2/detection_loss6.252628400325775 % _timestamp1.7884653672223554e+09 .loss/FRt4/detection_loss2.906523928593058 _step11 /loss/FRt2/assignment_loss6.722870534466159 /loss/FRt4/assignment_loss6.644566947167073 2loss/SRqqt2/assignment_loss3.3239664852619173 0loss/FBt3/assignment_loss0.4814481046155379 0loss/SRqqt3/detection_loss4.366701137854477 -loss/FBt3/detection_loss9.87962006404996 trainer/global_step149 /loss/FRt3/assignment_loss6.681273091407049 2loss/SRqqt4/assignment_loss3.1281101335497463 0loss/FBt4/assignment_loss0.6589301313622855 -loss/FRt3/detection_loss3.75477008823128 .loss/FRt1/detection_loss2.237507115961529 1loss/SRqqt2/detection_loss3.9306075792759656 epoch0 0loss/FRt1/assignment_loss6.6271338537922055 2loss/SRqqt1/assignment_loss3.2242353978364364S /loss/FRt2/assignment_loss6.722870534466159 /loss/FRt4/assignment_loss6.644566947167073 2loss/SRqqt2/assignment_loss3.3239664852619173 0loss/FBt3/assignment_loss0.4814481046155379 0loss/SRqqt3/detection_loss4.366701137854477 -loss/FBt3/detection_loss9.87962006404996 trainer/global_step149 /loss/FRt3/assignment_loss6.681273091407049 2loss/SRqqt4/assignment_loss3.1281101335497463 0loss/FBt4/assignment_loss0.6589301313622855 -loss/FRt3/detection_loss3.75477008823128 .loss/FRt1/detection_loss2.237507115961529 1loss/SRqqt2/detection_loss3.9306075792759656 epoch0 0loss/FRt1/assignment_loss6.6271338537922055 2loss/SRqqt1/assignment_loss3.2242353978364364 1loss/FBt2/assignment_loss0.49779384104695057 .loss/FRt2/detection_loss3.599359381583429 1loss/SRqqt4/detection_loss2.1840746219215146 0loss/SRqqt1/detection_loss2.198344301711487 /loss/FBt1/detection_loss2.8172087023163788 /loss/FBt4/detection_loss7.2707676630450635 %loss/total_loss93.58928491635456 _runtime 257.262828404 1loss/SRqqt3/assignment_loss3.867203722397486 0loss/FBt1/assignment_loss0.3336396885817953 .loss/FBt2/detection_loss6.252628400325775 % _timestamp1.7884653672223554e+09 .loss/FRt4/detection_loss2.906523928593058 _step11 he(:  network.sent271053 network.recv150053 disk.dm-1.in0 disk.dm-1.out8.890625 disk.sda3.out8.890625 disk.sda.in0 disk.sda2.in0 disk.sda2.out0 disk.drbd1260.out 0.03515625" disk.dm-0.out5352.08056640625 disk.sda1.in0 disk.sda.out8.890625" disk.nvme0c0n1.in 603.25390625' disk.nvme0c0n1.out5352.08056640625( proc.memory.availableMB 4517.4921875# proc.memory.rssMB 3019.80078125( disk./.usagePercent76.8680862093288 disk.drbd1260.in0 disk.nvme0n1.in 603.25390625% disk.nvme0n1.out5352.08056640625 disk.sda3.in0% disk./.usageGB1145.3900337219238 disk.dm-0.in 603.25390625 disk.sda1.out0$ memory_percent72.42741584777832* proc.memory.percent18.431401252746582 cpu21.546437315710023 proc.cpu.threads172?T(:  gpu.0.gpu97.0 gpu.0.memory23+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp44.0 gpu.0.powerWatts119.037' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.8074 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0׃6(:  gpu.0.gpu97.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts331.951' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.3902 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0|d(: 똂 disk.dm-1.out 9.42578125 disk.sda1.out0 disk.sda2.in0 proc.cpu.threads17 network.sent309006 network.recv154915) disk./.usagePercent76.90165186915812 disk.drbd1260.in0 disk.dm-0.in 715.2109375" disk.dm-0.out5508.24365234375 disk.sda3.in0# proc.memory.rssMB 3019.80078125 disk.dm-1.in0 disk.nvme0n1.in 715.2109375 disk.sda1.in0 disk.sda3.out 9.42578125 disk.sda.in0 disk.sda.out 9.42578125$ memory_percent72.42927551269531* proc.memory.percent18.431401252746582! disk.nvme0c0n1.in 715.2109375 disk.drbd1260.out 0.0390625% disk.nvme0n1.out5508.24365234375 disk.sda2.out0% proc.memory.availableMB 4517.1875 cpu21.70356573588438% disk./.usageGB1145.8901863098145' disk.nvme0c0n1.out5508.24365234375o(:  gpu.0.gpu84.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts209.169' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent41.833800000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0e(: 廃 disk.sda3.in0 disk.drbd1260.out 0.0390625 disk.dm-0.in 715.2109375# proc.memory.rssMB 3019.80078125 cpu21.84529553989264 proc.cpu.threads17) disk./.usagePercent76.90165058912044! disk.nvme0c0n1.in 715.2109375$ memory_percent72.42259979248047& proc.memory.availableMB 4518.28125* proc.memory.percent18.431401252746582 disk.dm-1.in0 network.sent313587 disk.sda3.out 9.66796875 disk.sda.in0 disk.sda.out 9.66796875 disk.sda2.in0 disk.sda2.out0 disk.drbd1260.in0 network.recv159419' disk.nvme0c0n1.out5662.08740234375 disk.dm-1.out 9.66796875 disk.nvme0n1.in 715.2109375% disk.nvme0n1.out5662.08740234375" disk.dm-0.out5662.08740234375 disk.sda1.in0 disk.sda1.out0% disk./.usageGB1145.8901672363281[(:  network.recv164001 disk.dm-0.out 6010.171875 disk.sda2.out0" disk.nvme0c0n1.out 6010.171875 disk.dm-1.in0' proc.memory.availableMB 4518.265625% disk./.usageGB1145.8901481628418! disk.nvme0c0n1.in 715.2109375 disk.dm-1.out 10.09765625 disk.drbd1260.out 0.04296875$ memory_percent72.42269515991211 disk.sda.out 10.09765625 disk.nvme0n1.in 715.2109375 disk.nvme0n1.out 6010.171875 disk.sda1.in0 disk.sda1.out0 disk.sda3.out 10.09765625* proc.memory.percent18.431401252746582 cpu21.98034944443662) disk./.usagePercent76.90164930908277 disk.dm-0.in 715.2109375 disk.sda3.in0 disk.sda2.in0 disk.drbd1260.in0# proc.memory.rssMB 3019.80078125 proc.cpu.threads17 disk.sda.in0 network.sent31936399(:  gpu.0.gpu72.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts217.688' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent43.5376 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0} (: ١ gpu.0.gpu76.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts235.64' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.128 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0uJ(: ) disk./.usagePercent76.90161782015605 disk.sda2.out0! disk.nvme0c0n1.in 715.2109375! disk.nvme0n1.out 6392.3984375 disk.dm-0.out 6392.3984375 disk.sda3.out 10.6015625% disk./.usageGB1145.8896789550781# disk.nvme0c0n1.out 6392.3984375 disk.drbd1260.out 0.04296875 disk.nvme0n1.in 715.2109375 disk.dm-0.in 715.2109375$ memory_percent72.42424488067627 cpu22.103413291983184 disk.dm-1.in0 disk.dm-1.out 10.6015625 disk.sda.in0 disk.sda.out 10.6015625# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 disk.sda2.in0 disk.drbd1260.in0 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0) proc.memory.availableMB 4518.01171875 proc.cpu.threads17 network.sent325501 network.recv168772Z0+ _wandbruntime339  _runtime339:)S & lr-AdamW/pg10.0003880347568936332 & lr-AdamW/pg20.0003880347568936332 trainer/global_step199 $ _timestamp1.788465449242513e+09 _runtime 339.283216813 _step126\t $ _timestamp1.788465449242513e+09 _runtime 339.283216813 _step12 & lr-AdamW/pg10.0003880347568936332 & lr-AdamW/pg20.0003880347568936332 trainer/global_step199 G0+ _wandbruntime339  _runtime339/,'O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated2783556 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated 9618771505152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _step13 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed874142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 587094712832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed42714459838976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed3680248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed42260344054784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 456631484928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated3698638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed2783550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated42717000967680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current84505600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _runtime 339.285180095 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed6463798 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 587039305216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed2783550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 9031676792320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated2783556 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed40573804842480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated42260369482752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 454115784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated3698638 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed41027324044404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated1368249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed1367783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 454115784192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated2242394 % _timestamp1.7884654492441044e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 453519201924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated40573830270448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed42260344054784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed 9618686999552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 trainer/global_step199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed42714459838976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated42260369482752 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed42714459838976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed3680248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed42260344054784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 456631484928 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated3698638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed2783550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated42717000967680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current84505600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _runtime 339.285180095 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed6463798 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 587039305216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed2783550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated 9031676792320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated2783556 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed40573804842480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated42260369482752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 454115784192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated3698638 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed41027324044404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated1368249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed1367783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 454115784192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated2242394 % _timestamp1.7884654492441044e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 453519201924 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated40573830270448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed42260344054784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed 9618686999552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 trainer/global_step199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed42714459838976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated42260369482752 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 HDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed2797431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated40776358980032 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 458305819074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed6496030 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated878477 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1375046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 455791456778 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated6514452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed3698599 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed41232125008842 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed6496030 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed2797431 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated42471359858176 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 9665764873216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed42471334430208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed2253067 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 458906714624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated2797437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed40776333552064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated6514452 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 590026711552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 _step14 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 456391000576 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed42927725430784 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated2797437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 % _timestamp1.7884654508226082e+09 _runtime 340.863816888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated42930266572800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated3717015 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 589971317248U\M VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 XGDeviceStatq!sMonitor.on_train_batch_end/active_bytes.small_pool.allocated 458906714624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated2797437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed40776333552064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated6514452 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 590026711552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 _step14 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 456391000576 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed42927725430784 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated2797437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 % _timestamp1.7884654508226082e+09 _runtime 340.863816888 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated42930266572800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated3717015 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 589971317248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed2797431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated40776358980032 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 458305819074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed6496030 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated878477 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1375046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 455791456778 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated6514452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed3698599 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed41232125008842 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed6496030 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed2797431 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated42471359858176 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated 9665764873216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed42471334430208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed2253067 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264AҚ0+ _wandbruntime340  _runtime340Ã2K $loss/total_loss86.0487384723934 epoch0 /loss/FBt2/assignment_loss0.536286378154263 .loss/FRt1/assignment_loss6.71300498789007 0loss/SRqqt4/detection_loss2.218927261263132 -loss/FBt2/detection_loss4.91479730416428 /loss/FBt3/detection_loss7.7252572549240925 trainer/global_step199 /loss/FRt2/detection_loss2.9314615726470947 0loss/FBt3/assignment_loss0.3854091734517819 0loss/SRqqt2/detection_loss4.574105327412234 /loss/FRt4/assignment_loss7.395188715550806 1loss/SRqqt3/assignment_loss2.767008474469185 1loss/SRqqt4/assignment_loss3.097269037246704 .loss/FRt4/detection_loss3.903724735820448 1loss/SRqqt1/detection_loss1.3236896845016155 % _timestamp1.7884654508253338e+09 _step15 /loss/FRt1/detection_loss1.8215903854731357 /loss/FRt3/assignment_loss6.885738601147289 0loss/FBt1/assignment_loss0.4018762035678364 /loss/FRt3/detection_loss4.0160133947788825 1loss/SRqqt3/detection_loss3.6031691443175076 .loss/FBt4/detection_loss6.194123592641619 _runtime 340.865730937 /loss/FRt2/assignment_loss6.822109899213237 1loss/SRqqt1/assignment_loss2.744170681742214 1loss/SRqqt2/assignment_loss2.958142401808399 0loss/FBt4/assignment_loss0.6408488055070242 /loss/FBt1/detection_loss1.4748254547005366O 0loss/SRqqt4/detection_loss2.218927261263132 -loss/FBt2/detection_loss4.91479730416428 /loss/FBt3/detection_loss7.7252572549240925 trainer/global_step199 /loss/FRt2/detection_loss2.9314615726470947 0loss/FBt3/assignment_loss0.3854091734517819 0loss/SRqqt2/detection_loss4.574105327412234 /loss/FRt4/assignment_loss7.395188715550806 1loss/SRqqt3/assignment_loss2.767008474469185 1loss/SRqqt4/assignment_loss3.097269037246704 .loss/FRt4/detection_loss3.903724735820448 1loss/SRqqt1/detection_loss1.3236896845016155 % _timestamp1.7884654508253338e+09 _step15 /loss/FRt1/detection_loss1.8215903854731357 /loss/FRt3/assignment_loss6.885738601147289 0loss/FBt1/assignment_loss0.4018762035678364 /loss/FRt3/detection_loss4.0160133947788825 1loss/SRqqt3/detection_loss3.6031691443175076 .loss/FBt4/detection_loss6.194123592641619 _runtime 340.865730937 /loss/FRt2/assignment_loss6.822109899213237 1loss/SRqqt1/assignment_loss2.744170681742214 1loss/SRqqt2/assignment_loss2.958142401808399 0loss/FBt4/assignment_loss0.6408488055070242 /loss/FBt1/detection_loss1.4748254547005366 $loss/total_loss86.0487384723934 epoch0 /loss/FBt2/assignment_loss0.536286378154263 .loss/FRt1/assignment_loss6.713004987890079ch(:  disk.sda.out 11.0234375 disk.sda2.out0 disk.dm-1.out 11.0234375$ memory_percent72.42357730865479) proc.memory.availableMB 4518.12109375* proc.memory.percent18.431401252746582 network.sent330933 disk.dm-0.in 803.45703125 disk.sda3.in0 proc.cpu.threads17 network.recv172419 disk.sda2.in0" disk.nvme0c0n1.in 803.45703125' disk.nvme0c0n1.out6728.89501953125% disk.nvme0n1.out6728.89501953125 disk.sda3.out 11.0234375# proc.memory.rssMB 3019.80078125 cpu22.220164793755245) disk./.usagePercent76.90004286180542 disk.sda.in0 disk.dm-1.in0 disk.drbd1260.in0 disk.nvme0n1.in 803.45703125" disk.dm-0.out6728.89501953125 disk.sda1.in0 disk.sda1.out0 disk.drbd1260.out0.046875" disk./.usageGB1145.8662109375 wّ(:  gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts324.468' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.8936 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors03Kh(: ' disk.nvme0c0n1.out7123.10986328125 disk.drbd1260.in0 disk.sda.in0# proc.memory.rssMB 3019.80078125 cpu22.328146625107372 proc.cpu.threads17 network.recv178667% disk./.usageGB1145.8896789550781! disk.nvme0c0n1.in 866.1953125% disk.nvme0n1.out7123.10986328125 disk.dm-0.in 866.1953125 disk.sda3.out 11.4296875 disk.sda.out 11.4296875 disk.sda2.in0) disk./.usagePercent76.90161782015605 disk.dm-1.in0 disk.sda2.out0$ memory_percent72.42376804351807* proc.memory.percent18.431401252746582" disk.dm-0.out7123.10986328125 disk.dm-1.out 11.4296875 disk.drbd1260.out0.046875 disk.nvme0n1.in 866.1953125 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0) proc.memory.availableMB 4518.08984375 network.sent381251v;b(:  gpu.0.gpu97.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts121.684' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent24.3368 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0SG(: ̢ gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp41.0 gpu.0.powerWatts115.943' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.1886 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0,j~e(: ̢ disk.sda2.in0 disk.sda2.out0& disk.nvme0c0n1.out7446.5556640625 disk.drbd1260.in0% disk./.usageGB1145.8896980285645! disk.nvme0c0n1.in 866.1953125 disk.dm-1.out 11.8046875 disk.nvme0n1.in 866.1953125 disk.dm-0.in 866.1953125 disk.sda1.out0 disk.sda3.out 11.8046875$ memory_percent72.42536544799805$ disk.nvme0n1.out7446.5556640625! disk.dm-0.out7446.5556640625' proc.memory.availableMB 4517.828125* proc.memory.percent18.431401252746582 cpu22.428972913878447 network.recv183249 disk.dm-1.in0 disk.drbd1260.out 0.05078125 disk.sda1.in0 disk.sda3.in0# proc.memory.rssMB 3019.80078125 proc.cpu.threads17 network.sent387100) disk./.usagePercent76.90161910019373 disk.sda.in0 disk.sda.out 11.80468751 (: ۢ gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts242.846' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.5692 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0d(: ۢ network.recv187810! disk.nvme0c0n1.in 866.1953125 disk.dm-1.out 12.2578125 disk.drbd1260.in0# proc.memory.rssMB 3019.80078125 cpu22.523216042689118 disk.sda3.out 12.2578125) disk./.usagePercent76.90161782015605# disk.nvme0n1.out7600.470703125 disk.dm-0.in 866.1953125 disk.dm-0.out7600.470703125 disk.sda3.in0% disk.nvme0c0n1.out7600.470703125 disk.dm-1.in0 network.sent393189% disk./.usageGB1145.8896789550781 disk.sda.out 12.2578125 disk.sda2.in0 disk.sda2.out0 disk.drbd1260.out 0.05078125$ memory_percent72.42410182952881) proc.memory.availableMB 4518.03515625 disk.sda.in0 disk.nvme0n1.in 866.1953125 disk.sda1.in0 disk.sda1.out0* proc.memory.percent18.431401252746582 proc.cpu.threads17Jr1(:  gpu.0.gpu37.0 gpu.0.memory9+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts331.578' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.3156 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0jOi(:  network.sent398569% disk.nvme0n1.out7853.31591796875 disk.sda1.in0 disk.sda2.in0 disk.dm-1.in0 disk.dm-1.out 12.94921875 disk.drbd1260.in0# proc.memory.rssMB 3019.80078125 disk.sda.out 12.94921875 disk.sda2.out0! disk.nvme0c0n1.in 866.1953125$ memory_percent72.42422103881836* proc.memory.percent18.431401252746582 cpu22.61333542720725 proc.cpu.threads17 network.recv191843% disk./.usageGB1145.8898086547852 disk.nvme0n1.in 866.1953125" disk.dm-0.out7853.31591796875 disk.sda3.in0 disk.sda3.out 12.94921875' disk.nvme0c0n1.out7853.31591796875' proc.memory.availableMB 4518.015625) disk./.usagePercent76.90162652441222 disk.dm-0.in 866.1953125 disk.sda1.out0 disk.sda.in0 disk.drbd1260.out 0.0546875 (:  gpu.0.gpu67.0 gpu.0.memory11+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts237.269' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.4538 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 hk(: ߲ disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 13.23046875 disk.sda2.out0! disk.nvme0c0n1.in 1005.171875% disk.nvme0n1.out8151.42919921875" disk.dm-0.out8151.42919921875# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582( disk./.usagePercent76.9000502860239 disk.sda.in0 disk.sda2.in0' disk.nvme0c0n1.out8151.42919921875 disk.drbd1260.out 0.0546875 disk.dm-0.in 1005.171875 cpu22.696850359241925 network.sent404925 disk.dm-1.in0 disk.dm-1.out 13.23046875 disk.drbd1260.in0$ memory_percent72.42071628570557) proc.memory.availableMB 4518.58984375 proc.cpu.threads17 network.recv196613% disk./.usageGB1145.8663215637207 disk.sda.out 13.23046875 disk.nvme0n1.in 1005.171875b|0+ _wandbruntime422  _runtime422  _runtime 422.61646347 _step16 ' lr-AdamW/pg10.00048553092696741035 ' lr-AdamW/pg20.00048553092696741035 trainer/global_step249 % _timestamp1.7884655325693579e+09 ' lr-AdamW/pg10.00048553092696741035 ' lr-AdamW/pg20.00048553092696741035 trainer/global_step249 % _timestamp1.7884655325693579e+09 _runtime 422.61646347 _step16anM0+ _wandbruntime422  _runtime422['O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed52809862825984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed53377767257600 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed2798419 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 trainer/global_step249 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 570420132352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed4601150 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated11234745259520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated51269939228368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated52809888253952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current447 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 569673478720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated50700265749648 % _timestamp1.7884655325771954e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated3477606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed11968204254720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 570420132352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated1708124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current82408448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated2798869 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 _runtime 422.618395953 _step17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed3477600 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 733488093184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed8078750 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 567904431616 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed4601150 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated3477606 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated11968286663168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated52809888253952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed11234716161536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed50700240321680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated53380308386304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed3477600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 733488093184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed8078750 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 567904431616 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed4601150 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated3477606 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated11968286663168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated52809888253952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_+>split_bytes.large_pool.freed11234716161536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed50700240321680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed52809862825984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed53377767257600 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed2798419 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 trainer/global_step249 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 570420132352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed4601150 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated11234745259520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated51269939228368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated52809888253952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current447 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 569673478720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated50700265749648 % _timestamp1.7884655325771954e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated3477606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed11968204254720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 570420132352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated1708124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current82408448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated2798869 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 _runtime 422.618395953 _step17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed3477600 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated4637917 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 570179648000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed12015195594240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 570179648000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed1714515 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed50902769031264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed53020853201408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 569431371486 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated3491487 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated53020878629376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1714971 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed4619501 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed2809589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed8110982 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed3491481 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current84492288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 % _timestamp1.7884655340734248e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated8129404 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed8110982 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated11278806628864 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 571945733782 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 572695362048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1095074 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated53593573991424 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated4637917 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 trainer/global_step249 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed53591032849408 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated8129404 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated53020878629376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 736418063360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed51472200402750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated8129404 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated53020878629376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 736418063360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed51472200402750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated4637917 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 570179648000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed12015195594240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current55394304 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 570179648000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed1714515 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed50902769031264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed53020853201408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 569431371486 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated3491487 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated53020878629376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated1714971 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed4619501 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed2809589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed8110982 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed3491481 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current84492288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141ugu0+ _wandbruntime424  _runtime424eP /loss/FRt4/assignment_loss6.061127285162608 1loss/SRqqt2/assignment_loss2.734985653559367 2loss/SRqqt3/assignment_loss2.8362292372263394 2loss/SRqqt4/assignment_loss2.6831686739029923 /loss/FRt4/detection_loss2.8498501290256777 1loss/SRqqt2/detection_loss3.6145371705293656 _runtime 424.116073203 1loss/SRqqt1/assignment_loss2.726550677243401 0loss/FBt3/assignment_loss0.3746399127408545 0loss/FBt4/assignment_loss0.4147156143758016 /loss/FRt1/detection_loss2.0785935309761805 /loss/FRt2/assignment_loss6.774069583777226 /loss/FRt3/assignment_loss6.499313259124756 1loss/SRqqt3/detection_loss3.9671024554050884 .loss/FBt2/detection_loss5.449249624166378 trainer/global_step249 $ _timestamp1.788465534075721e+09 0loss/FBt2/assignment_loss0.5425970247482372 0loss/SRqqt4/detection_loss2.162400791436676 .loss/FBt3/detection_loss10.86428522070249 .loss/FBt4/detection_loss4.849083166324354 _step19 0loss/SRqqt1/detection_loss0.824717392718109 %loss/total_loss81.37733539599006 epoch0 /loss/FRt1/assignment_loss4.470782445677629 /loss/FRt3/detection_loss2.9655954079495537 0loss/FBt1/assignment_loss0.3173864617234936 /loss/FRt2/detection_loss3.1132549318400295 /loss/FBt1/detection_loss2.2030997456534513WDT 0loss/FBt1/assignment_loss0.3173864617234936 /loss/FRt2/detection_loss3.1132549318400295 /loss/FBt1/detection_loss2.2030997456534513 /loss/FRt4/assignment_loss6.061127285162608 1loss/SRqqt2/assignment_loss2.734985653559367 2loss/SRqqt3/assignment_loss2.8362292372263394 2loss/SRqqt4/assignment_loss2.6831686739029923 /loss/FRt4/detection_loss2.8498501290256777 1loss/SRqqt2/detection_loss3.6145371705293656 _runtime 424.116073203 1loss/SRqqt1/assignment_loss2.726550677243401 0loss/FBt3/assignment_loss0.3746399127408545 0loss/FBt4/assignment_loss0.4147156143758016 /loss/FRt1/detection_loss2.0785935309761805 /loss/FRt2/assignment_loss6.774069583777226 /loss/FRt3/assignment_loss6.499313259124756 1loss/SRqqt3/detection_loss3.9671024554050884 .loss/FBt2/detection_loss5.449249624166378 trainer/global_step249 $ _timestamp1.788465534075721e+09 0loss/FBt2/assignment_loss0.5425970247482372 0loss/SRqqt4/detection_loss2.162400791436676 .loss/FBt3/detection_loss10.86428522070249 .loss/FBt4/detection_loss4.849083166324354 _step19 0loss/SRqqt1/detection_loss0.824717392718109 %loss/total_loss81.37733539599006 epoch0 /loss/FRt1/assignment_loss4.470782445677629 /loss/FRt3/detection_loss2.9655954079495537!DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated64043615804928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated1307345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed14317809851392 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated63359407025152 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 681693079040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 % _timestamp1.7884656148881245e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed5522052 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed4171650 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed64041074676224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated4171656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated5540442 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated14317894356992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated60826701228848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 681693079040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _runtime 504.92925535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed64041074676224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated2049363 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 880025222656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed61507474832220 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 trainer/global_step299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current55407616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 _step21 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated4171656 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated3356708 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed4171650 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed2048907 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456㷎'O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed63359381597184 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed9693702 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated9712098 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 880080630272 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 680799031340 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed13437784628736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated63359407025152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated13437813726720 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated61510014622276 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 684208779776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed5522052 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated64043615804928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated1307345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed14317809851392 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated63359407025152 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 681693079040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 % _timestamp1.7884656148881245e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed5522052 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed4171650 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed64041074676224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated4171656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated5540442 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated14317894356992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated60826701228848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 681693079040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 _runtime 504.92925535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed64041074676224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated2049363 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 880025222656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed61507474832220 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 trainer/global_step299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current55407616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 _step21 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated4171656 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated3356708 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed4171650 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed2048907 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed1307342 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed3356249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328kf0+ _wandbruntime506  _runtime506Æ M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 $ _timestamp1.788465616458334e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated4185537 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated5558819 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated64256881410048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 _step22 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated63570397400576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed63570371972608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1311674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated63570397400576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current436 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated4185537 trainer/global_step299 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated9744356 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed3367295 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 883002354688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 683071286194 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed61712275796658 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated9744356 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed5540403 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 683968295424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed14364795055616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 _runtime 506.499629446 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 686484009472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed64254340268032 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed61029204510464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 685585648490 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed9725934 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated14364877450752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated3367734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 882949057536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672ȷL&M M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed61029204510464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 685585648490 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed9725934 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated14364877450752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated3367734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 882949057536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 $ _timestamp1.788465616458334e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated4185537 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated5558819 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated64256881410048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 _step22 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated63570397400576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed63570371972608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1311674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated63570397400576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current436 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated4185537 trainer/global_step299 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated9744356 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed3367295 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 883002354688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 683071286194 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed61712275796658 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated9744356 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed5540403 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 683968295424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed14364795055616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 _runtime 506.499629446 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 686484009472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed64254340268032{0+ _wandbruntime506  _runtime506j`R /loss/FRt4/assignment_loss5.780083258946736 _step23 /loss/FRt2/assignment_loss6.384174652588674 1loss/SRqqt3/detection_loss3.4403407610952854 .loss/FBt3/detection_loss3.374833854960232 2loss/SRqqt4/assignment_loss2.5602894118151713 0loss/FBt1/assignment_loss0.4167972855966245 0loss/FBt2/assignment_loss0.3292387775973361 2loss/SRqqt2/assignment_loss2.8271090626716613 0loss/FBt3/assignment_loss0.5376426432644501 0loss/SRqqt2/detection_loss3.596825885027647 1loss/SRqqt4/detection_loss2.5272265623525247 /loss/FBt1/detection_loss1.9139782507297272 /loss/FBt2/detection_loss3.5073218869942204 %loss/total_loss72.65549075023115 1loss/SRqqt1/detection_loss0.8547403596652051 .loss/FBt4/detection_loss7.659320525825024 1loss/SRqqt3/assignment_loss3.476541519165039 /loss/FRt1/detection_loss1.9257610946893693 .loss/FRt4/detection_loss2.591728162330886 epoch0 _runtime 506.500996053 /loss/FRt1/assignment_loss3.428798151899267 2loss/SRqqt1/assignment_loss2.5733150916440146 0loss/FBt4/assignment_loss0.5078967666485497 /loss/FRt3/detection_loss2.8709499876793116 trainer/global_step299 /loss/FRt3/assignment_loss6.364960350402414 .loss/FRt2/detection_loss3.205616446641775 % _timestamp1.7884656164605875e+09LV /loss/FBt2/detection_loss3.5073218869942204 %loss/total_loss72.65549075023115 1loss/SRqqt1/detection_loss0.8547403596652051 .loss/FBt4/detection_loss7.659320525825024 1loss/SRqqt3/assignment_loss3.476541519165039 /loss/FRt1/detection_loss1.9257610946893693 .loss/FRt4/detection_loss2.591728162330886 epoch0 _runtime 506.500996053 /loss/FRt1/assignment_loss3.428798151899267 2loss/SRqqt1/assignment_loss2.5733150916440146 0loss/FBt4/assignment_loss0.5078967666485497 /loss/FRt3/detection_loss2.8709499876793116 trainer/global_step299 /loss/FRt3/assignment_loss6.364960350402414 .loss/FRt2/detection_loss3.205616446641775 % _timestamp1.7884656164605875e+09 /loss/FRt4/assignment_loss5.780083258946736 _step23 /loss/FRt2/assignment_loss6.384174652588674 1loss/SRqqt3/detection_loss3.4403407610952854 .loss/FBt3/detection_loss3.374833854960232 2loss/SRqqt4/assignment_loss2.5602894118151713 0loss/FBt1/assignment_loss0.4167972855966245 0loss/FBt2/assignment_loss0.3292387775973361 2loss/SRqqt2/assignment_loss2.8271090626716613 0loss/FBt3/assignment_loss0.5376426432644501 0loss/SRqqt2/detection_loss3.596825885027647 1loss/SRqqt4/detection_loss2.5272265623525247 /loss/FBt1/detection_loss1.9139782507297272ٶ(: ӣބ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts237.803' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.560599999999994 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors04(q(: ӣ disk.drbd1260.in0% disk.nvme0n1.out9941.31005859375 disk.dm-0.in 1168.78515625 disk.sda.out 16.01171875 disk.sda2.out0# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582# disk.nvme0c0n1.in 1168.78515625 disk.dm-1.out 16.01171875 disk.drbd1260.out 0.0703125 disk.sda3.out 16.01171875 disk.sda.in0 disk.sda2.in0 proc.cpu.threads17' disk.nvme0c0n1.out9941.31005859375! disk.nvme0n1.in 1168.78515625 disk.sda1.in0 disk.sda3.in0 cpu23.10892961492956 network.sent497648 network.recv224784) disk./.usagePercent76.83012182804617% disk./.usageGB1144.8243370056152" disk.dm-0.out9941.31005859375 disk.sda1.out0$ memory_percent72.41995334625244) proc.memory.availableMB 4518.71484375 disk.dm-1.in0*(:  gpu.0.gpu91.0 gpu.0.memory21+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts334.788' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.9576 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0$p(:  disk.drbd1260.in0 disk.drbd1260.out 0.07421875) disk./.usagePercent76.82854558965785 disk.sda1.out0 disk.sda3.in0 disk.sda.in0# disk.nvme0c0n1.in 1168.78515625# disk.dm-0.out10229.21435546875$ memory_percent72.42002487182617# proc.memory.rssMB 3019.80078125 network.sent534112 disk.sda.out 16.4765625( disk.nvme0c0n1.out10229.21435546875! disk.nvme0n1.in 1168.78515625 disk.dm-0.in 1168.78515625' proc.memory.availableMB 4518.703125 disk.sda1.in0 disk.dm-1.out 16.4765625& disk.nvme0n1.out10229.21435546875* proc.memory.percent18.431401252746582 cpu23.16620908342701 proc.cpu.threads17 network.recv230223% disk./.usageGB1144.8008499145508 disk.sda3.out 16.4765625 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0ys(: # proc.memory.rssMB 3019.80078125( disk./.usagePercent76.8217555018272 disk.dm-1.in0& disk.nvme0n1.out10615.03466796875 disk.dm-0.in 1309.51171875 disk.sda1.out0* proc.memory.percent18.431401252746582 proc.cpu.threads17 network.recv234669 disk.drbd1260.out 0.07421875 disk.sda.in0 disk.sda2.out0( disk.nvme0c0n1.out10615.03466796875% disk./.usageGB1144.6996726989746 disk.dm-1.out 16.95703125 disk.sda3.out 16.95703125 disk.sda2.in0# disk.nvme0c0n1.in 1309.51171875$ memory_percent72.42002487182617' proc.memory.availableMB 4518.703125 cpu23.221559905666588 network.sent539552 disk.drbd1260.in0! disk.nvme0n1.in 1309.51171875# disk.dm-0.out10615.03466796875 disk.sda1.in0 disk.sda3.in0 disk.sda.out 16.95703125L'(:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp42.0 gpu.0.powerWatts119.857' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.971400000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0y $(:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp49.0 gpu.0.powerWatts337.98' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.596 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Kc(:  disk.dm-1.out17.34375 disk.drbd1260.out0.078125$ memory_percent72.42012023925781! disk.nvme0n1.in 1309.51171875# disk.nvme0c0n1.in 1309.51171875' disk.nvme0c0n1.out10891.0341796875* proc.memory.percent18.431401252746582 cpu23.27233414223946 proc.cpu.threads17 disk.sda3.in0 disk.sda.in0 network.sent545330 network.recv239181) disk./.usagePercent76.82878751677771% disk./.usageGB1144.8044548034668% disk.nvme0n1.out10891.0341796875 disk.dm-0.in 1309.51171875 disk.sda1.in0 disk.sda3.out17.34375" disk.dm-0.out10891.0341796875 disk.sda2.in0 disk.sda2.out0 disk.drbd1260.in0% proc.memory.availableMB 4518.6875# proc.memory.rssMB 3019.80078125 disk.sda.out17.34375 disk.sda1.out0 disk.dm-1.in0c&mo(: Տ* proc.memory.percent18.431401252746582) disk./.usagePercent76.83423689315156$ disk./.usageGB1144.885654449463% disk.nvme0n1.out11286.9091796875 disk.sda1.in0 disk.sda3.in0 disk.dm-1.out 17.71484375 proc.cpu.threads17 network.sent553101 network.recv243519 disk.dm-0.in 1309.51171875 disk.sda3.out 17.71484375 disk.sda.in0 disk.sda.out 17.71484375 disk.sda2.out0' disk.nvme0c0n1.out11286.9091796875! disk.nvme0n1.in 1309.51171875# disk.nvme0c0n1.in 1309.51171875 disk.dm-1.in0 disk.drbd1260.in0$ memory_percent72.42007255554199( proc.memory.availableMB 4518.6953125# proc.memory.rssMB 3019.80078125 cpu23.321603931522706" disk.dm-0.out11286.9091796875 disk.sda1.out0 disk.sda2.in0 disk.drbd1260.out0.078125<(: ޯ gpu.0.gpu41.0 gpu.0.memory10+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp49.0 gpu.0.powerWatts339.32' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.864 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0%(: ΀ gpu.0.gpu93.0 gpu.0.memory22+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts323.123' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.6246 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0}n(: # disk.dm-0.out11383.25341796875 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0 network.recv247472) disk./.usagePercent76.83266782297419 disk.dm-1.out 18.171875 disk.sda3.in0 disk.sda.out 18.171875( proc.memory.availableMB 4518.6953125# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 disk.sda3.out 18.171875 disk.sda2.out0 cpu23.36981531219947 proc.cpu.threads17 disk.dm-0.in 1309.51171875 network.sent558313 disk.drbd1260.in0 disk.drbd1260.out 0.08203125! disk.nvme0n1.in 1309.51171875& disk.nvme0n1.out11383.25341796875 disk.sda.in0$ memory_percent72.42007255554199% disk./.usageGB1144.8622741699219# disk.nvme0c0n1.in 1309.51171875( disk.nvme0c0n1.out11383.25341796875 disk.dm-1.in00+ _wandbruntime587  _runtime5877(E % _timestamp1.7884656976986716e+09 _runtime 587.739168392 _step24 & lr-AdamW/pg10.0006805232671149647 & lr-AdamW/pg20.0006805232671149647 trainer/global_step349m$ trainer/global_step349 % _timestamp1.7884656976986716e+09 _runtime 587.739168392 _step24 & lr-AdamW/pg10.0006805232671149647 & lr-AdamW/pg20.0006805232671149647n-G0+ _wandbruntime587  _runtime587`]7a&O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated74706923223552 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 797997427200 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 795481726464 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 795481726464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed73908900368384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated73908925796352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated73908925796352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed4865700 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated70953136708048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed74704382094848 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated6461344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1026726101504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed1523942 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed74704382094848 _runtime 587.743199202 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated4865706 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated16667608295424 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed4865700 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated74706923223552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed2389867 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated15640882193920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 796953308136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated1523945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed73908900368384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1026672791040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 797997427200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed3913809 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed71747550226128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated2390305 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 794438946048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed6442954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed74704382094848 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated6461344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1026726101504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed1523942 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed74704382094848 _runtime 587.743199202 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated4865706 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated16667608295424 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed4865700 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated74706923223552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed2389867 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated15640882193920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 796953308136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated1523945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed73908900368384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1026672791040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 797997427200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed3913809 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed71747550226128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated2390305 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 794438946048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed6442954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 797997427200 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 795481726464 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 795481726464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed73908900368384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated73908925796352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated73908925796352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed4865700 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated70953136708048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated4879587 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 800272656896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated6479721 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated6479721 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 % _timestamp1.7884656993128636e+09 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated74920188828672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 797756942848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed3924769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 800272656896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated2396946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated71155665417632 _runtime 589.354338136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed11340886 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed6461305 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated3925223 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1029590897152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1029644194304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated11359308 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed71155639989664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step349 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed11340886 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed16714505362432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0z'&M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed2396495 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated16714587757568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current454 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed15684914465280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated71954890980830 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed71952351190566 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated74920188828672 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed6461305 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed4879581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step349 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed11340886 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed16714505362432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 799225563198 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed4879581 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated74119916171776 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 797756942848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated4879587 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 800272656896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated6479721 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated6479721 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 % _timestamp1.7884656993128636e+09 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated74920188828672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 797756942848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed3924769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 800272656896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated2396946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated71155665417632 _runtime 589.354338136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed11340886 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed6461305 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated3925223 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1029590897152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1029644194304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated11359308 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed71155639989664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 Ef(: ڤ disk.sda3.in0 disk.sda3.out 19.734375 disk.sda.out 19.734375' disk.nvme0c0n1.out12534.4814453125 disk.dm-1.in0% disk.nvme0n1.out12534.4814453125 disk.dm-0.in 1518.0859375 disk.sda1.out0 network.sent623516 network.recv266019) disk./.usagePercent76.77035584514765$ disk./.usageGB1143.933780670166 disk.sda.in0 disk.sda2.out0" disk.nvme0c0n1.in 1518.0859375" disk.dm-0.out12534.4814453125 disk.nvme0n1.in 1518.0859375 disk.sda1.in0$ memory_percent72.42071628570557) proc.memory.availableMB 4518.58984375# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 cpu23.535424292076723 disk.sda2.in0 disk.dm-1.out 19.734375 disk.drbd1260.in0 disk.drbd1260.out0.09375 proc.cpu.threads17U#(:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts226.537' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.3074 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0u/׷c(: # memory_percent72.4189281463623# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582" disk.nvme0c0n1.in 1518.0859375 disk.nvme0n1.in 1518.0859375" disk.dm-0.out12593.3642578125 disk.sda3.in0 disk.sda.in0 disk.sda2.out0 cpu23.572110639686( disk./.usagePercent76.7578708697178 disk.dm-1.out 20.0546875% disk.nvme0n1.out12593.3642578125 disk.dm-0.in 1518.0859375 disk.sda1.in0 disk.sda.out 20.0546875 disk.sda2.in0 proc.cpu.threads17 network.sent629549$ disk./.usageGB1143.747745513916 disk.drbd1260.in0 disk.sda3.out 20.0546875( proc.memory.availableMB 4518.8828125 network.recv270751' disk.nvme0c0n1.out12593.3642578125 disk.dm-1.in0 disk.drbd1260.out0.09375 disk.sda1.out0we'0+ _wandbruntime670  _runtime670djv $ _timestamp1.788465780374822e+09 _runtime 670.415465973 _step28 & lr-AdamW/pg10.0007780194371887419 & lr-AdamW/pg20.0007780194371887419 trainer/global_step399) & lr-AdamW/pg10.0007780194371887419 & lr-AdamW/pg20.0007780194371887419 trainer/global_step399 $ _timestamp1.788465780374822e+09 _runtime 670.415465973 _step28xj0+ _wandbruntime670  _runtime670">8z'O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed1740542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1173310801408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 910593222844 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated5559756 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed81987625620036 $ _timestamp1.788465780376549e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed85367689513472 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 909270373888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 trainer/global_step399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated85370230642176 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated84458444567552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 909270373888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed4471343 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed5559750 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed7363856 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 911786074624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current457 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated17843950661120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 trainer/global_step399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated85370230642176 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated84458444567552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 909270373888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed4471343 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed5559750 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed7363856 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 911786074624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current457 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated17843950661120 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 910593222844 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated5559756 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed81987625620036 $ _timestamp1.788465780376549e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed85367689513472 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 909270373888 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1744874 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 914061304320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed12955838 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated4483013 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated5573637 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 trainer/global_step399 _step30 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed12955838 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 911545590272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated12974260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 _runtime 671.959966892 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed7382207 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated84669434942976 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 914061304320 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 911545590272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed7382207 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated81282100896832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated1744877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated82194966374738 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated7400623 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 % _timestamp1.7884657819178524e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed17887982932480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1176282283008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated7400623 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed85580955105280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed4482554 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed84669409515008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated19064294313472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed2737680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed81282075468864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 359017912322&M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 % _timestamp1.7884657819178524e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed17887982932480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1176282283008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated7400623 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed85580955105280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed4482554 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed84669409515008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated19064294313472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed2737680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed81282075468864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated5573637 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed84669409515008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed19064211918336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1744874 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 914061304320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed12955838 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated4483013 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated5573637 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 trainer/global_step399 _step30 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed12955838 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 911545590272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated12974260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 _runtime 671.959966892 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed7382207 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated84669434942976 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 914061304320 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 911545590272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed7382207 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated81282100896832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated1744877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated82194966374738 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated7400623 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388Ir9p0+ _wandbruntime671  _runtime671U /loss/FRt4/assignment_loss4.068028471790827 0loss/FBt4/assignment_loss0.8970956972667149 _runtime 671.962865407 /loss/FRt4/detection_loss2.3800151032897143 % _timestamp1.7884657819220333e+09 /loss/FRt2/detection_loss2.6001086099238337 /loss/FRt3/detection_loss2.9132025114753666 /loss/FBt1/detection_loss0.9669272650748344 /loss/FBt2/detection_loss2.9840880643862944 .loss/FBt3/detection_loss4.361347241835161 &loss/total_loss60.148697736888515 /loss/FRt1/assignment_loss2.351984588412551 2loss/SRqqt4/assignment_loss2.0626760735398246 1loss/SRqqt3/detection_loss2.7459032899803586 .loss/FBt4/detection_loss5.378557469163622 2loss/SRqqt1/assignment_loss2.0755309087913587 2loss/SRqqt2/assignment_loss2.3150935841458185 0loss/SRqqt2/detection_loss2.227552682161331 /loss/FRt1/detection_loss1.9791922306155298 1loss/SRqqt1/detection_loss1.0964822685398625 0loss/SRqqt4/detection_loss2.044812781328247 trainer/global_step399 _step31 epoch0 /loss/FRt2/assignment_loss4.799459186535847 0loss/SRqqt3/assignment_loss2.85126052548488 1loss/FBt1/assignment_loss0.27058346830799995 1loss/FBt3/assignment_loss0.47360787622293093 /loss/FRt3/assignment_loss5.687753500307307 0loss/FBt2/assignment_loss0.6174343383082976zxY /loss/FRt1/assignment_loss2.351984588412551 2loss/SRqqt4/assignment_loss2.0626760735398246 1loss/SRqqt3/detection_loss2.7459032899803586 .loss/FBt4/detection_loss5.378557469163622 2loss/SRqqt1/assignment_loss2.0755309087913587 2loss/SRqqt2/assignment_loss2.3150935841458185 0loss/SRqqt2/detection_loss2.227552682161331 /loss/FRt1/detection_loss1.9791922306155298 1loss/SRqqt1/detection_loss1.0964822685398625 0loss/SRqqt4/detection_loss2.044812781328247 trainer/global_step399 _step31 epoch0 /loss/FRt2/assignment_loss4.799459186535847 0loss/SRqqt3/assignment_loss2.85126052548488 1loss/FBt1/assignment_loss0.27058346830799995 1loss/FBt3/assignment_loss0.47360787622293093 /loss/FRt3/assignment_loss5.687753500307307 0loss/FBt2/assignment_loss0.6174343383082976 /loss/FRt4/assignment_loss4.068028471790827 0loss/FBt4/assignment_loss0.8970956972667149 _runtime 671.962865407 /loss/FRt4/detection_loss2.3800151032897143 % _timestamp1.7884657819220333e+09 /loss/FRt2/detection_loss2.6001086099238337 /loss/FRt3/detection_loss2.9132025114753666 /loss/FBt1/detection_loss0.9669272650748344 /loss/FBt2/detection_loss2.9840880643862944 .loss/FBt3/detection_loss4.361347241835161 &loss/total_loss60.148697736888515k@O(:  disk.dm-1.in0 disk.drbd1260.out0.09375$ memory_percent72.41935729980469 disk.nvme0n1.in 1639.109375% disk.nvme0n1.out12834.1005859375 disk.dm-0.in 1639.109375 disk.sda1.in0 disk.sda.out20.5 disk.sda2.in0# proc.memory.rssMB 3019.80078125 proc.cpu.threads17 disk.sda3.out20.5 disk.sda2.out0! disk.nvme0c0n1.in 1639.109375' disk.nvme0c0n1.out12834.1005859375 disk.dm-1.out20.5% proc.memory.availableMB 4518.8125 cpu23.60804505070702$ disk./.usageGB1143.747787475586 disk.sda3.in0 disk.sda.in0 disk.drbd1260.in0* proc.memory.percent18.431401252746582 network.sent635250 network.recv275065) disk./.usagePercent76.75787368580067" disk.dm-0.out12834.1005859375 disk.sda1.out0c (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts217.449' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent43.4898 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0a m(: $ memory_percent72.41942882537842) proc.memory.availableMB 4518.80078125# proc.memory.rssMB 3019.80078125 network.recv280450% disk./.usageGB1143.9743003845215" disk.dm-0.out13073.1904296875 disk.sda3.out 21.20703125' disk.nvme0c0n1.out13073.1904296875 disk.dm-1.out 21.20703125* proc.memory.percent18.431401252746582 proc.cpu.threads17) disk./.usagePercent76.77307515717634 disk.dm-0.in 1639.109375 disk.sda1.in0! disk.nvme0c0n1.in 1639.109375 disk.drbd1260.out 0.09765625 cpu23.642288970992837 disk.sda.in0 disk.sda.out 21.20703125 network.sent684764% disk.nvme0n1.out13073.1904296875 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0 disk.dm-1.in0 disk.nvme0n1.in 1639.109375 disk.sda2.out0 disk.drbd1260.in0fd(: ٣ gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp41.0 gpu.0.powerWatts117.827' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.5654 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0qbd(:  disk.dm-0.out13422.08203125 disk.sda2.out0$ memory_percent72.42097854614258 cpu23.67592918997905 proc.cpu.threads17 network.recv284469% disk.nvme0c0n1.out13422.08203125 disk.dm-1.out 21.55859375 disk.drbd1260.out 0.09765625 disk.nvme0n1.in 1639.109375# disk.nvme0n1.out13422.08203125 disk.sda1.in0 disk.sda3.out 21.55859375% disk./.usageGB1143.9977416992188! disk.nvme0c0n1.in 1639.109375 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda2.in0' proc.memory.availableMB 4518.546875* proc.memory.percent18.431401252746582 network.sent689975) disk./.usagePercent76.77464832347425 disk.sda.out 21.55859375# proc.memory.rssMB 3019.80078125 disk.dm-1.in0 disk.drbd1260.in0 disk.dm-0.in 1639.109375(: ʿ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts211.006' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent42.2012 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0#(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts250.052' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent50.0104 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0k+a(: ܂ network.sent697971) disk./.usagePercent76.77464832347425 disk.nvme0n1.in 1639.109375 disk.sda1.out0 disk.sda2.in0 disk.sda2.out0* proc.memory.percent18.431401252746582 disk.drbd1260.out 0.1015625# memory_percent72.4196195602417) proc.memory.availableMB 4518.76953125 proc.cpu.threads17 disk.dm-0.out13673.18359375 network.recv289230! disk.nvme0c0n1.in 1639.109375 disk.dm-1.in0 disk.dm-1.out 22.3828125 disk.drbd1260.in0 disk.sda3.in0 disk.sda.in0% disk./.usageGB1143.9977416992188% disk.nvme0c0n1.out13673.18359375 disk.dm-0.in 1639.109375 disk.sda1.in0 disk.sda3.out 22.3828125 disk.sda.out 22.3828125# proc.memory.rssMB 3019.80078125 cpu23.70807637462677# disk.nvme0n1.out13673.18359375e(:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts207.581' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent41.5162 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0$ܟs(:  disk.dm-1.in0# disk.dm-0.out13852.45751953125 disk.sda.in0 disk.sda.out 22.82421875 cpu23.739117509370264( disk.nvme0c0n1.out13852.45751953125 disk.dm-1.out 22.82421875& disk.nvme0n1.out13852.45751953125 disk.sda1.in0 disk.sda2.out0$ memory_percent72.41964340209961# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582% disk./.usageGB1143.9977188110352 disk.drbd1260.in0! disk.nvme0n1.in 1763.43359375 disk.dm-0.in 1763.43359375 disk.sda3.out 22.82421875' proc.memory.availableMB 4518.765625 proc.cpu.threads17 network.recv293376) disk./.usagePercent76.77464678742903 disk.drbd1260.out 0.1015625 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0 network.sent701720# disk.nvme0c0n1.in 1763.43359375q)Or(: å disk.sda1.out0 disk.dm-1.in0 disk.dm-1.out 23.1953125) proc.memory.availableMB 4518.74609375% disk./.usageGB1144.3353805541992! disk.nvme0n1.in 1763.43359375 disk.dm-0.in 1763.43359375 disk.sda3.in0 disk.sda3.out 23.1953125 disk.sda.in0 disk.sda2.in0( disk.nvme0c0n1.out14009.95068359375 network.sent707229 disk.sda.out 23.1953125 disk.drbd1260.in0 disk.drbd1260.out 0.10546875$ memory_percent72.41976261138916# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 cpu23.76944549849226& disk.nvme0n1.out14009.95068359375 disk.sda2.out0# disk.nvme0c0n1.in 1763.43359375 proc.cpu.threads17 network.recv297624) disk./.usagePercent76.79730755032985# disk.dm-0.out14009.95068359375 disk.sda1.in0Նo(: å gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts236.142' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.2284 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0CJ?0+ _wandbruntime753  _runtime753s _step32 & lr-AdamW/pg10.0008755156072625191 & lr-AdamW/pg20.0008755156072625191 trainer/global_step449 % _timestamp1.7884658632703238e+09 _runtime 753.311122124d# trainer/global_step449 % _timestamp1.7884658632703238e+09 _runtime 753.311122124 _step32 & lr-AdamW/pg10.0008755156072625191 & lr-AdamW/pg20.0008755156072625191 W0+ _wandbruntime753  _runtime753DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated14556954 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed8284758 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed5028548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed92227701013944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated96033538060800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1021718775464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1319950705152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated91206007666448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 trainer/global_step449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed96030996932096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed95007937910784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1025574722048 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated95007963338752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed14538558 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated5029001 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed6253800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current450 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3071406 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed91205982238480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed08W'O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 % _timestamp1.7884658632710686e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated14556954 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed8284758 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed5028548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed92227701013944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated96033538060800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1021718775464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1319950705152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated91206007666448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 trainer/global_step449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed96030996932096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed95007937910784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1025574722048 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated95007963338752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed14538558 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated5029001 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed6253800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current450 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3071406 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed91205982238480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed96030996932096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed20046990030336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096!>(A0+ _wandbruntime754  _runtime754YO&M ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated14589212 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed96244262523904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated1961477 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated14589212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed14570790 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed8303109 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated95218953714176 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated8321525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed8303109 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated3078758 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated20091080497664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed21413882432512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed5039800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1026505392614 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current435 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1322831032832 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated92435041768646 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1023991030318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1027849951744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 % _timestamp1.7884658647478228e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated96246803665920 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated96246803665920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed14570790 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated8321525 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed20091051399680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated91408536376032 _runtime 754.78863046 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1322884329984 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated6267687 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated95218953714176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1025334237696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1961474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated6267687 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed6267681 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed6267681 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated21413964827648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current367241#M aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1322884329984 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated6267687 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated95218953714176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1025334237696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed1961474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated6267687 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed6267681 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed6267681 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated21413964827648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated14589212 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed96244262523904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated1961477 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated14589212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed14570790 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed8303109 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated95218953714176 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated8321525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed8303109 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated3078758 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated20091080497664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed21413882432512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed5039800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1026505392614 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current435 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1322831032832 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated92435041768646 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1023991030318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1027849951744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 % _timestamp1.7884658647478228e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated96246803665920 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated96246803665920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed14570790 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated8321525 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed20091051399680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated91408536376032 _runtime 754.78863046"d60+ _wandbruntime754  _runtime7547TU 2loss/SRqqt1/assignment_loss1.5997385770703356 /loss/FRt2/assignment_loss4.422461848258973 0loss/FRt3/assignment_loss5.3036469357912654 1loss/SRqqt2/detection_loss2.0987998127289442 2loss/SRqqt4/assignment_loss1.8524328610249634 0loss/FBt1/assignment_loss0.2735728217109921 /loss/FRt2/detection_loss2.3783695274591445 1loss/SRqqt3/detection_loss2.6379596252333033 1loss/SRqqt4/detection_loss1.8309859674552391 .loss/FBt4/detection_loss3.833660738808768 epoch0 _runtime 754.790096273 0loss/FBt4/assignment_loss0.6941399787153516 /loss/FRt1/detection_loss1.8287199804825442 /loss/FRt4/detection_loss2.3320639816302697 .loss/FBt2/detection_loss3.408389261364937 %loss/total_loss54.03715331641252 .loss/FRt4/assignment_loss3.72722107924304 1loss/SRqqt1/detection_loss1.2343823214371998 /loss/FBt3/detection_loss3.0808174389859904 0loss/FBt2/assignment_loss0.5317867320962251 /loss/FBt3/assignment_loss0.501408526107021 /loss/FBt1/detection_loss0.8702760699191756 _step35 % _timestamp1.7884658647497199e+09 2loss/SRqqt2/assignment_loss2.1266915176225747 /loss/FRt3/detection_loss2.9809632340415577 0loss/FRt1/assignment_loss1.8356565194470542 1loss/SRqqt3/assignment_loss2.653007959777659 trainer/global_step4490Y 2loss/SRqqt1/assignment_loss1.5997385770703356 /loss/FRt2/assignment_loss4.422461848258973 0loss/FRt3/assignment_loss5.3036469357912654 1loss/SRqqt2/detection_loss2.0987998127289442 2loss/SRqqt4/assignment_loss1.8524328610249634 0loss/FBt1/assignment_loss0.2735728217109921 /loss/FRt2/detection_loss2.3783695274591445 1loss/SRqqt3/detection_loss2.6379596252333033 1loss/SRqqt4/detection_loss1.8309859674552391 .loss/FBt4/detection_loss3.833660738808768 epoch0 _runtime 754.790096273 0loss/FBt4/assignment_loss0.6941399787153516 /loss/FRt1/detection_loss1.8287199804825442 /loss/FRt4/detection_loss2.3320639816302697 .loss/FBt2/detection_loss3.408389261364937 %loss/total_loss54.03715331641252 .loss/FRt4/assignment_loss3.72722107924304 1loss/SRqqt1/detection_loss1.2343823214371998 /loss/FBt3/detection_loss3.0808174389859904 0loss/FBt2/assignment_loss0.5317867320962251 /loss/FBt3/assignment_loss0.501408526107021 /loss/FBt1/detection_loss0.8702760699191756 _step35 % _timestamp1.7884658647497199e+09 2loss/SRqqt2/assignment_loss2.1266915176225747 /loss/FRt3/detection_loss2.9809632340415577 0loss/FRt1/assignment_loss1.8356565194470542 1loss/SRqqt3/assignment_loss2.653007959777659 trainer/global_step449#s(: ҥ# disk.dm-0.out14189.14990234375 disk.sda1.in0 disk.sda1.out0 disk.sda3.out 23.66015625 disk.sda.in0 disk.sda2.in0# disk.nvme0c0n1.in 1763.43359375( disk.nvme0c0n1.out14189.14990234375 network.sent712254) disk./.usagePercent76.79563812519908 disk.sda3.in0 disk.sda.out 23.66015625 disk.sda2.out0 disk.dm-1.in0$ memory_percent72.42071628570557# proc.memory.rssMB 3019.80078125# disk./.usageGB1144.31050491333& disk.nvme0n1.out14189.14990234375 disk.dm-1.out 23.66015625 disk.drbd1260.out 0.10546875) proc.memory.availableMB 4518.58984375* proc.memory.percent18.431401252746582 cpu23.79808128479426 proc.cpu.threads17 network.recv300931! disk.nvme0n1.in 1763.43359375 disk.dm-0.in 1763.43359375 disk.drbd1260.in0SP(: ҥ gpu.0.gpu46.0 gpu.0.memory11+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts117.402' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.480400000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 j(:  disk.drbd1260.in0 disk.drbd1260.out0.109375* proc.memory.percent18.431401252746582 cpu23.82553567652882! disk.nvme0n1.in 1763.43359375 disk.sda1.in0 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0$ memory_percent72.42200374603271 disk.sda.in0 disk.sda.out 24.1328125& disk.nvme0c0n1.out14441.556640625) proc.memory.availableMB 4518.37890625# proc.memory.rssMB 3019.80078125 proc.cpu.threads17 network.sent762083 network.recv307206) disk./.usagePercent76.79723868430314 disk.dm-0.in 1763.43359375! disk.dm-0.out14441.556640625 disk.sda1.out0 disk.sda3.out 24.1328125# disk.nvme0c0n1.in 1763.43359375% disk./.usageGB1144.3343544006348$ disk.nvme0n1.out14441.556640625 disk.dm-1.in0 disk.dm-1.out 24.1328125ɦp(:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts243.911' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent48.782199999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors06(:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts328.93' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.786 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0f-k(: $ disk./.usageGB1144.357307434082 disk.drbd1260.in0 disk.drbd1260.out0.109375! disk.nvme0n1.in 1932.49609375 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.dm-1.out 24.70703125) disk./.usagePercent76.79877908163667 disk.dm-0.in 1932.49609375! disk.dm-0.out14716.751953125& disk.nvme0c0n1.out14716.751953125 disk.dm-1.in0# memory_percent72.4236011505127# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 network.recv310965$ disk.nvme0n1.out14716.751953125 disk.sda3.in0 disk.sda.out 24.70703125# disk.nvme0c0n1.in 1932.49609375( proc.memory.availableMB 4518.1171875 cpu23.853968740010146 network.sent767601 disk.sda3.out 24.70703125 disk.sda2.in0 disk.sda2.out0 proc.cpu.threads178}Mr(: ݨ& disk.nvme0n1.out15017.27197265625 disk.sda3.in0$ memory_percent72.42331504821777* proc.memory.percent18.431401252746582 network.recv314958) disk./.usagePercent76.79877780159899 disk.dm-1.in0 disk.sda1.out0 disk.sda.in0 disk.sda2.in0( disk.nvme0c0n1.out15017.27197265625 disk.dm-1.out 25.2734375 disk.dm-0.in 1932.49609375# proc.memory.rssMB 3019.80078125 cpu23.880955531603178 proc.cpu.threads17 disk.sda2.out0# disk.nvme0c0n1.in 1932.49609375 disk.drbd1260.in0 disk.drbd1260.out 0.11328125# disk.dm-0.out15017.27197265625 disk.sda1.in0 disk.sda3.out 25.2734375( proc.memory.availableMB 4518.1640625 network.sent773308% disk./.usageGB1144.3572883605957 disk.sda.out 25.2734375! disk.nvme0n1.in 1932.49609375(:  gpu.0.gpu44.0 gpu.0.memory11+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp48.0 gpu.0.powerWatts325.434' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.0868 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0i,w(:  gpu.0.gpu16.0 gpu.0.memory4+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp42.0 gpu.0.powerWatts153.433' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent30.6866 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0sOu(: 㫤 disk.drbd1260.in0# proc.memory.rssMB 3019.80078125& disk.nvme0n1.out15244.42041015625 disk.dm-0.in 1932.49609375 disk.sda1.in0# disk.nvme0c0n1.in 1932.49609375 disk.dm-1.out 25.63671875 disk.drbd1260.out 0.11328125$ memory_percent72.42333889007568 proc.cpu.threads17) disk./.usagePercent76.79720284324834 disk.sda.in0) proc.memory.availableMB 4518.16015625* proc.memory.percent18.431401252746582 cpu23.90686749099581 disk.sda.out 25.63671875 network.recv318352% disk./.usageGB1144.3338203430176# disk.dm-0.out15244.42041015625 disk.sda1.out0 disk.sda3.in0 disk.sda2.out0( disk.nvme0c0n1.out15244.42041015625 network.sent778355! disk.nvme0n1.in 1932.49609375 disk.sda3.out 25.63671875 disk.sda2.in0 disk.dm-1.in0P0W 0+ _wandbruntime836  _runtime836ub & lr-AdamW/pg10.0009730117773362962 & lr-AdamW/pg20.0009730117773362962 trainer/global_step499 % _timestamp1.7884659460344799e+09 _runtime 836.077177463 _step36^UH & lr-AdamW/pg10.0009730117773362962 & lr-AdamW/pg20.0009730117773362962 trainer/global_step499 % _timestamp1.7884659460344799e+09 _runtime 836.077177463 _step36$*I0+ _wandbruntime836  _runtime8360*K'O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1136847668736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 $ _timestamp1.788465946035676e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed9205660 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated106696845479424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed102467776407852 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed101332417717680 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed2173742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2173745 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed22250058497536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1466483373568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1137873052260 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated9224050 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed105557456681984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated16171906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current469 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed9205660 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated23716570969088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1135358690172 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated102470316197908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed106694304350720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 _runtime 836.077884817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated6947856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated105557482109952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated22250087595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 _step37 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated106696845479424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3410574 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated101332443145648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated105557482109952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed5584316 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated3411040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current86602752 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed106694304350720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed23716484366336 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed6947850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated102470316197908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed106694304350720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 _runtime 836.077884817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated6947856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated105557482109952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated22250087595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 _step37 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated106696845479424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3410574 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated101332443145648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated105557482109952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed5584316 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated3411040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current86602752 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed106694304350720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed23716484366336 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed6947850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed5595501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1141638599168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1139122885120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated6961737 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed105768447057408 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated5595955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed6961731 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated16204164 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2178074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current57491456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed102672577372290 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1141638599168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated3417878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated22294148964864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 trainer/global_step499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated106910111084544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed106907569942528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current451 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated9242427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated101534971855232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated105768472485376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1139122885120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed16185742 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1469350259200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated102675117162554 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed101534946427264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed16185742 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1469407750656 _step38 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated6961737 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed6961731 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed9224011 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated105768472485376 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated9242427 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed23763470126080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed22294119866880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated2178077 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed105768447057408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _runtime 837.626661417 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _runtime 837.626661417 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed5595501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539790264 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1141638599168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1139122885120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated6961737 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed105768447057408 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated5595955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed6961731 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated16204164 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2178074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current57491456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed102672577372290 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1141638599168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated3417878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142090752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated22294148964864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 trainer/global_step499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated106910111084544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed106907569942528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current451 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated9242427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated101534971855232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated105768472485376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1139122885120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed16185742 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current86589440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1469350259200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated102675117162554 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed101534946427264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed16185742 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1469407750656 _step38 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated6961737 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed6961731 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed9224011 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated105768472485376 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated9242427 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed23763470126080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed22294119866880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated2178077&Du0+ _wandbruntime837  _runtime837"T epoch0 $ _timestamp1.788465947588361e+09 _step39 2loss/SRqqt1/assignment_loss1.4649775381773138 /loss/FBt3/detection_loss3.4354464496885027 1loss/FBt1/assignment_loss0.25071710889893095 /loss/FBt1/detection_loss0.9147160009751527 .loss/FBt2/detection_loss3.315263638893763 0loss/FBt3/assignment_loss0.6972586000753965 /loss/FRt2/assignment_loss4.679653832989354 /loss/FRt3/assignment_loss5.107208440178319 2loss/SRqqt3/assignment_loss2.2144737550571785 2loss/SRqqt4/assignment_loss2.1128645912573045 /loss/FRt4/detection_loss2.4676144236012507 _runtime 837.628830656 /loss/FRt1/detection_loss1.9538647978713637 0loss/SRqqt3/detection_loss2.590826798286011 0loss/SRqqt4/detection_loss2.062716916203499 $loss/total_loss54.2315446896089 1loss/SRqqt2/detection_loss1.8305598361925646 /loss/FRt3/detection_loss3.1167213874950743 1loss/SRqqt1/detection_loss1.4267260826550998 .loss/FBt4/detection_loss3.019535710414251 0loss/FBt2/assignment_loss0.7758090805469288 0loss/FBt4/assignment_loss0.3553083616674745 /loss/FRt1/assignment_loss2.081972210030807 /loss/FRt2/detection_loss2.3049314332905637 trainer/global_step499 0loss/FRt4/assignment_loss3.9308822895351208 2loss/SRqqt2/assignment_loss2.1214954056276762SpiX $ _timestamp1.788465947588361e+09 _step39 2loss/SRqqt1/assignment_loss1.4649775381773138 /loss/FBt3/detection_loss3.4354464496885027 1loss/FBt1/assignment_loss0.25071710889893095 /loss/FBt1/detection_loss0.9147160009751527 .loss/FBt2/detection_loss3.315263638893763 0loss/FBt3/assignment_loss0.6972586000753965 /loss/FRt2/assignment_loss4.679653832989354 /loss/FRt3/assignment_loss5.107208440178319 2loss/SRqqt3/assignment_loss2.2144737550571785 2loss/SRqqt4/assignment_loss2.1128645912573045 /loss/FRt4/detection_loss2.4676144236012507 _runtime 837.628830656 /loss/FRt1/detection_loss1.9538647978713637 0loss/SRqqt3/detection_loss2.590826798286011 0loss/SRqqt4/detection_loss2.062716916203499 $loss/total_loss54.2315446896089 1loss/SRqqt2/detection_loss1.8305598361925646 /loss/FRt3/detection_loss3.1167213874950743 1loss/SRqqt1/detection_loss1.4267260826550998 .loss/FBt4/detection_loss3.019535710414251 0loss/FBt2/assignment_loss0.7758090805469288 0loss/FBt4/assignment_loss0.3553083616674745 /loss/FRt1/assignment_loss2.081972210030807 /loss/FRt2/detection_loss2.3049314332905637 trainer/global_step499 0loss/FRt4/assignment_loss3.9308822895351208 2loss/SRqqt2/assignment_loss2.1214954056276762 epoch0'Dcy(:  gpu.0.gpu65.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts212.352' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent42.470400000000005 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors06c(:  disk.sda.in0$ memory_percent72.42507934570312* proc.memory.percent18.431401252746582 cpu23.931722246994088! disk.dm-0.out15400.955078125$ disk.nvme0n1.out15400.955078125 disk.dm-0.in 1932.49609375 disk.sda3.in0 disk.sda.out 26.140625& disk.nvme0c0n1.out15400.955078125# proc.memory.rssMB 3019.80078125 proc.cpu.threads17$ disk./.usageGB1144.357521057129! disk.nvme0n1.in 1932.49609375 disk.sda1.in0 disk.sda2.out0# disk.nvme0c0n1.in 1932.49609375$ proc.memory.availableMB4517.875 disk.sda1.out0 network.sent783556 network.recv322077) disk./.usagePercent76.79879341805858 disk.dm-1.out 26.140625 disk.drbd1260.in0 disk.drbd1260.out 0.1171875 disk.sda2.in0 disk.dm-1.in0 disk.sda3.out 26.140625p^(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts247.462' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.492399999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0rki(: 辬 network.sent832920 disk.sda2.out0 disk.drbd1260.out 0.1171875 disk.dm-0.in 2097.57421875 disk.sda3.in0 disk.sda.in0) proc.memory.availableMB 4518.01953125% disk./.usageGB1143.9295387268066# disk.nvme0c0n1.in 2097.57421875% disk.nvme0c0n1.out15532.20703125 disk.dm-1.out 26.4921875# disk.nvme0n1.out15532.20703125 disk.sda1.out0 disk.sda3.out 26.4921875 disk.sda.out 26.4921875 disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.in0 disk.dm-0.out15532.20703125$ memory_percent72.42419719696045# proc.memory.rssMB 3019.80078125 cpu23.955698760404072 proc.cpu.threads17 network.recv327130) disk./.usagePercent76.77007116476956! disk.nvme0n1.in 2097.57421875 disk.sda1.in0* proc.memory.percent18.431401252746582avr(:  disk.dm-1.in0 disk.dm-1.out 26.85546875) proc.memory.availableMB 4518.46484375$ disk./.usageGB1144.084617614746! disk.nvme0n1.in 2097.57421875 disk.sda2.in0 disk.drbd1260.in0 disk.drbd1260.out 0.12109375# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 cpu23.979426180756565 disk.sda3.in0 disk.sda.out 26.85546875$ memory_percent72.42147922515869 proc.cpu.threads17 disk.sda3.out 26.85546875 network.sent838548 network.recv331396% disk.nvme0n1.out15955.8505859375 disk.dm-0.in 2097.57421875" disk.dm-0.out15955.8505859375 disk.sda1.out0# disk.nvme0c0n1.in 2097.57421875) disk./.usagePercent76.78047863905904 disk.sda1.in0 disk.sda.in0 disk.sda2.out0' disk.nvme0c0n1.out15955.8505859375(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts255.961' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent51.1922 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Gu(: ʦǙ$ memory_percent72.42305278778076# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 cpu24.002580856807764 network.sent845512( disk./.usagePercent76.7909324507122% disk./.usageGB1144.2403869628906 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.dm-0.in 2097.57421875( disk.nvme0c0n1.out16147.08154296875 disk.dm-1.out 27.26171875) proc.memory.availableMB 4518.20703125 proc.cpu.threads17 disk.sda3.out 27.26171875 disk.sda.in0 disk.sda.out 27.26171875 network.recv334640! disk.nvme0n1.in 2097.57421875& disk.nvme0n1.out16147.08154296875 disk.sda1.in0# disk.nvme0c0n1.in 2097.57421875# disk.dm-0.out16147.08154296875 disk.sda1.out0 disk.drbd1260.in0 disk.drbd1260.out 0.12109375#(: ʦ gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts237.245' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.449000000000005 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors06)h(: ٦ disk.sda.in0& disk.nvme0c0n1.out16365.576171875( proc.memory.availableMB 4517.9453125! disk.dm-0.out16365.576171875 disk.sda1.in0 disk.sda1.out0 disk.sda.out 27.66796875 disk.sda2.out0# disk.nvme0c0n1.in 2097.57421875* proc.memory.percent18.431401252746582 cpu24.02486081950994 network.sent849697! disk.nvme0n1.in 2097.57421875 disk.dm-0.in 2097.57421875 disk.dm-1.in0 disk.drbd1260.out0.125$ memory_percent72.42465019226074# proc.memory.rssMB 3019.80078125 proc.cpu.threads17 network.recv339276% disk./.usageGB1144.2397727966309$ disk.nvme0n1.out16365.576171875 disk.sda3.in0 disk.sda3.out 27.66796875 disk.sda2.in0 disk.dm-1.out 27.66796875 disk.drbd1260.in0( disk./.usagePercent76.7908912334992I_(: ٦Ӛ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts229.246' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.8492 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors08(:  gpu.0.gpu63.0 gpu.0.memory11+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts236.435' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.287 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0f?c(:  proc.cpu.threads17 disk.sda3.in0( disk./.usagePercent76.7909001937629% disk./.usageGB1144.2399063110352 disk.nvme0n1.in 2198.453125 disk.dm-0.in 2198.453125# disk.dm-0.out16642.10400390625 disk.sda1.out0 disk.sda2.in0 disk.sda3.out 28.1015625 disk.sda2.out0( disk.nvme0c0n1.out16642.10400390625 disk.dm-1.out 28.1015625 disk.drbd1260.out0.125# proc.memory.rssMB 3019.80078125 cpu24.047069900070166 disk.sda.in0 network.sent855172& disk.nvme0n1.out16642.10400390625$ memory_percent72.42469787597656% proc.memory.availableMB 4517.9375* proc.memory.percent18.431401252746582 network.recv343566 disk.sda1.in0 disk.sda.out 28.1015625! disk.nvme0c0n1.in 2198.453125 disk.dm-1.in0 disk.drbd1260.in0(C0+ _wandbruntime919  _runtime919 & lr-AdamW/pg10.0010705079474100736 & lr-AdamW/pg20.0010705079474100736 trainer/global_step549 $ _timestamp1.788466029303798e+09 _runtime 919.344473735 _step40g & lr-AdamW/pg10.0010705079474100736 & lr-AdamW/pg20.0010705079474100736 trainer/global_step549 $ _timestamp1.788466029303798e+09 _runtime 919.344473735 _step40(Ru0+ _wandbruntime919  _runtime919} O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated3749788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated111458878624848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated116107000881152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1253152016896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1613060736000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1248998604880 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated17786858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3749350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 trainer/global_step549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1613007425536 _runtime 919.346492751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current82408448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated112710391591816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated26066216798720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed111458853196880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated10144952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 _step41 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1250636316160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated7641906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed117357611769344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2390345 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed26066134390272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated117360152898048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated116107000881152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed6139692 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed116106975453184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed116106975453184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed7641900 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1250636316160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated24453156062720 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed117357611769344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752sj'O OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed7641900 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1250636316160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated24453156062720 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed117357611769344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated3749788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current53310464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated111458878624848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated116107000881152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1253152016896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current18396 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1613060736000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1248998604880 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated17786858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed3749350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 trainer/global_step549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1613007425536 _runtime 919.346492751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2515700736 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current82408448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated112710391591816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated26066216798720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed111458853196880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated10144952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 _step41 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current18390 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140730052 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1250636316160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated7641906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed117357611769344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541128704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2390345 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed26066134390272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated117360152898048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated116107000881152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed6139692 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed116106975453184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed116106975453184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0)895 0+ _wandbruntime920  _runtime920.&M ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed116317965828608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated2394677 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed112912652766198 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak22687 _runtime 920.895329615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated17819116 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated7655787 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed7655781 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated24497217432064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed6150789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed10144913 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated117573418503168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 _step42 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated10163329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated112915192556462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated117573418503168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed24497188334080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed117570877361152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1615953379328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed26113141713408 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated6151228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1252911532544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated17819116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 % _timestamp1.7884660308541706e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated3756551 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current53297152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1255427246592 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak22388 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed17800694 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated10163329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2394674 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated26113224108544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed111661381906464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated116317991256576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1253785222030 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1255427246592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed10144913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed116317965828608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed17800694 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1353325056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated10163329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2394674 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated26113224108544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed111661381906464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current18416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated116317991256576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1253785222030 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1255427246592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed10144913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed116317965828608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed6150789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed10144913 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated117573418503168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 _step42 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated10163329 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated112915192556462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak22687 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated117573418503168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed24497188334080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142090752 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current82395136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed117570877361152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1615953379328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 409820160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed26113141713408 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2515714048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated6151228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1252911532544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated17819116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 % _timestamp1.7884660308541706e+09*$0+ _wandbruntime920  _runtime920H$ߖR 1loss/FBt1/assignment_loss0.21735495511921596 % _timestamp1.7884660308570783e+09 0loss/FBt4/assignment_loss0.1903198513105251 0loss/SRqqt4/detection_loss2.079836228619451 /loss/FBt1/detection_loss0.8032937915728585 1loss/SRqqt2/assignment_loss2.163646959752407 0loss/FBt3/assignment_loss0.4688730831503084 /loss/FRt4/detection_loss2.3804292559623716 .loss/FBt2/detection_loss3.610660043027666 epoch0 /loss/FRt3/assignment_loss4.813534544225325 2loss/SRqqt3/assignment_loss2.4335681191512517 2loss/SRqqt4/assignment_loss2.3535010869736257 -loss/FRt2/detection_loss2.18540954385318 1loss/SRqqt2/detection_loss1.9793908272711735 /loss/FBt3/detection_loss3.3611991217261865 /loss/FBt4/detection_loss2.5045650860537654 %loss/total_loss51.08397087338265 0loss/FRt1/assignment_loss1.7225734187910955 /loss/FRt2/assignment_loss4.497171247122335 2loss/SRqqt1/assignment_loss1.2109581466389097 .loss/FRt3/detection_loss3.042123120082052 0loss/SRqqt1/detection_loss0.952801681149544 0loss/FRt4/assignment_loss3.6451908757812097 trainer/global_step549 1loss/FBt2/assignment_loss0.39867466501891613 .loss/FRt1/detection_loss1.731701225740835 /loss/SRqqt3/detection_loss2.33719399528844 _runtime 920.897520274 _step43)=V 1loss/SRqqt2/assignment_loss2.163646959752407 0loss/FBt3/assignment_loss0.4688730831503084 /loss/FRt4/detection_loss2.3804292559623716 .loss/FBt2/detection_loss3.610660043027666 epoch0 /loss/FRt3/assignment_loss4.813534544225325 2loss/SRqqt3/assignment_loss2.4335681191512517 2loss/SRqqt4/assignment_loss2.3535010869736257 -loss/FRt2/detection_loss2.18540954385318 1loss/SRqqt2/detection_loss1.9793908272711735 /loss/FBt3/detection_loss3.3611991217261865 /loss/FBt4/detection_loss2.5045650860537654 %loss/total_loss51.08397087338265 0loss/FRt1/assignment_loss1.7225734187910955 /loss/FRt2/assignment_loss4.497171247122335 2loss/SRqqt1/assignment_loss1.2109581466389097 .loss/FRt3/detection_loss3.042123120082052 0loss/SRqqt1/detection_loss0.952801681149544 0loss/FRt4/assignment_loss3.6451908757812097 trainer/global_step549 1loss/FBt2/assignment_loss0.39867466501891613 .loss/FRt1/detection_loss1.731701225740835 /loss/SRqqt3/detection_loss2.33719399528844 _runtime 920.897520274 _step43 1loss/FBt1/assignment_loss0.21735495511921596 % _timestamp1.7884660308570783e+09 0loss/FBt4/assignment_loss0.1903198513105251 0loss/SRqqt4/detection_loss2.079836228619451 /loss/FBt1/detection_loss0.8032937915728585+Mc(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts244.965' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.993 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ϸnWh(:  disk.sda.out 28.49609375 disk.sda2.out0! disk.nvme0c0n1.in 2219.984375( proc.memory.availableMB 4518.1015625* proc.memory.percent18.431401252746582 network.sent860935) disk./.usagePercent76.74808139761599 disk.dm-1.out 28.49609375 disk.sda1.in0 disk.sda3.in0 disk.sda3.out 28.49609375 cpu24.06695439455213 proc.cpu.threads17% disk./.usageGB1143.6018753051758 disk.dm-1.in0 disk.nvme0n1.in 2219.984375 disk.sda1.out0# proc.memory.rssMB 3019.80078125 disk.drbd1260.in0$ disk.nvme0n1.out16920.181640625! disk.dm-0.out16920.181640625 disk.sda.in0 disk.sda2.in0& disk.nvme0c0n1.out16920.181640625$ memory_percent72.42369651794434 network.recv347809 disk.drbd1260.out 0.12890625 disk.dm-0.in 2219.984375Q`e(: # memory_percent72.4222183227539* proc.memory.percent18.431401252746582 proc.cpu.threads17% disk./.usageGB1143.6095657348633! disk.nvme0c0n1.in 2219.984375& proc.memory.availableMB 4518.34375# proc.memory.rssMB 3019.80078125 disk.dm-0.in 2219.984375! disk.dm-0.out16928.404296875 disk.sda1.out0 disk.sda3.out 28.81640625 disk.sda.out 28.81640625& disk.nvme0c0n1.out16928.404296875 disk.drbd1260.in0 disk.drbd1260.out 0.12890625 network.sent911569 disk.sda.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.out 28.81640625 disk.nvme0n1.in 2219.984375$ disk.nvme0n1.out16928.404296875 cpu24.08663042320994 network.recv352326) disk./.usagePercent76.74859750880502 disk.sda1.in0 disk.sda3.in0 disk.dm-1.in03(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts232.983' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.5966 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0X(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts247.886' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.5772 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0-d^(: Ӗ disk.drbd1260.in0 disk.drbd1260.out 0.1328125 disk.sda1.out0 disk.sda3.in0 disk.sda.out 29.26953125 network.sent915756% disk./.usageGB1143.6095886230469! disk.nvme0c0n1.in 2219.984375 disk.sda.in0 disk.sda2.out0( proc.memory.availableMB 4518.0703125# disk.nvme0c0n1.out 17218.578125 disk.dm-1.out 29.26953125! disk.nvme0n1.out 17218.578125 disk.dm-0.in 2219.984375 disk.dm-0.out 17218.578125 disk.sda1.in0 disk.sda3.out 29.26953125 disk.sda2.in0) disk./.usagePercent76.74859904485022 disk.nvme0n1.in 2219.984375$ memory_percent72.42388725280762# proc.memory.rssMB 3019.80078125* proc.memory.percent18.431401252746582 cpu24.10608898624691 proc.cpu.threads17 network.recv356612 disk.dm-1.in05 (:  gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts251.69' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent50.33800000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0*!_(:  disk.sda.out 29.67578125' proc.memory.availableMB 4517.796875* proc.memory.percent18.431425094604492 proc.cpu.threads17 network.sent921245 network.recv360384% disk./.usageGB1143.6095733642578 disk.dm-1.in0 disk.dm-1.out 29.67578125 disk.drbd1260.in0 disk.dm-0.in 2219.984375 disk.sda1.in0$ disk.nvme0c0n1.out 17267.0546875 disk.drbd1260.out 0.1328125" disk.nvme0n1.out 17267.0546875 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda2.in0$ memory_percent72.42555618286133( disk./.usagePercent76.7485980208201! disk.nvme0c0n1.in 2219.984375 disk.sda2.out0" proc.memory.rssMB 3019.8046875 cpu24.124557875588078 disk.nvme0n1.in 2219.984375 disk.dm-0.out 17267.0546875 disk.sda3.out 29.67578125] (: ӹ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts229.491' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.8982 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0mmme(:  disk.drbd1260.out 0.13671875 network.sent926929) disk./.usagePercent76.75701606056097 disk.dm-1.in0 disk.dm-1.out 30.2578125# disk.dm-0.out17364.82568359375 disk.sda3.out 30.2578125 disk.sda.out 30.2578125 disk.sda2.in0 disk.drbd1260.in0! disk.nvme0n1.in 2332.56640625& disk.nvme0n1.out17364.82568359375 disk.sda1.in0! memory_percent72.42431640625* proc.memory.percent18.431425094604492 proc.cpu.threads17 network.recv364702$ disk./.usageGB1143.735008239746 disk.sda2.out0# disk.nvme0c0n1.in 2332.56640625 disk.sda1.out0 proc.memory.availableMB4518" proc.memory.rssMB 3019.8046875 cpu24.143303326413907( disk.nvme0c0n1.out17364.82568359375 disk.dm-0.in 2332.56640625 disk.sda3.in0 disk.sda.in0]YV2- _wandbruntime1001  _runtime1001 & lr-AdamW/pg10.0011680041174838506 & lr-AdamW/pg20.0011680041174838506 trainer/global_step599 % _timestamp1.7884661117599835e+09 _runtime1001.800504081 _step44^'xJ & lr-AdamW/pg10.0011680041174838506 & lr-AdamW/pg20.0011680041174838506 trainer/global_step599 % _timestamp1.7884661117599835e+09 _runtime1001.800504081 _step44,).F2- _wandbruntime1001  _runtime10011\'O RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2606945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated128023460316672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed8335950 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated11065854 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1366940664320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated8335956 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated26656224529920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed121585288676080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed11047464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1362638519588 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated121585314104048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed126656494224384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1364424963584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed126656494224384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1366940664320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1364424963584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed28415791966720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed122947927195668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 % _timestamp1.7884661117612295e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated126656519652352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated126656519652352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed4088255 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1365152881676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 trainer/global_step599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated6695635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed19383414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1759596534784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1759647748096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed6695197 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated19401810 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated28415872278016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed128020919187968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated11065854 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated4088690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current80311296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed128020919187968^Y O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated28415872278016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed128020919187968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541128704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated11065854 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated4088690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current80311296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed128020919187968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539790056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak22687 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938005141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2606945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated128023460316672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed8335950 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142090752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current18396 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated11065854 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1366940664320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated8335956 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated26656224529920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed121585288676080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed11047464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1362638519588 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated121585314104048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed126656494224384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1364424963584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed126656494224384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1366940664320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1364424963584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514362088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed28415791966720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed122947927195668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 % _timestamp1.7884661117612295e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated126656519652352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak22687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated126656519652352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak22388 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed4088255 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2515700736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1365152881676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 trainer/global_step599 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current18390 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142090752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak22388 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated6695635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed19383414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 1759596534784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1759647748096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed6695197 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated19401810 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144--2- _wandbruntime1003  _runtime1003t`&M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated8349837 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated11084231 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1762592322048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated19434068 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed126867484599808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current444 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated4095410 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed4094969 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1366700179968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed6706243 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated8349837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1366700179968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed123152728160106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated19434068 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated128236725921792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed8349831 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated11084231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1762539024896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296t`&M M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541142016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated11084231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1762539024896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2515714048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514362296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated8349837 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated11084231 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 1762592322048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated19434068 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed126867484599808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current444 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated4095410 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed4094969 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938005141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1366700179968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current18416 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed6706243 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140730052 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541142016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current18422 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated8349837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1366700179968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed123152728160106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated19434068 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated128236725921792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed8349831.}!2- _wandbruntime1003  _runtime10033eNZ /loss/FRt4/assignment_loss3.666990719114741 1loss/FBt2/assignment_loss0.10045322584388876 .loss/FRt2/detection_loss2.091079476156405 /loss/FBt4/detection_loss2.4993760375415577 epoch0 _runtime1003.331263959 _step47 2loss/SRqqt1/assignment_loss1.0765299778897315 /loss/FRt4/detection_loss2.4041122794151306 trainer/global_step599 % _timestamp1.7884661132903872e+09 1loss/SRqqt2/assignment_loss1.846291193688238 /loss/FBt4/assignment_loss0.735863035380402 .loss/FRt3/detection_loss3.216787944034654 0loss/FRt2/assignment_loss3.6530197092464993 1loss/SRqqt2/detection_loss1.9287062718927324 /loss/FBt1/detection_loss0.7502651989704221 %loss/total_loss51.07173331686685 /loss/FBt2/detection_loss3.9267451550279344 /loss/FBt3/detection_loss3.2944064490935383 2loss/SRqqt4/assignment_loss2.0521671721308503 1loss/FBt1/assignment_loss0.22612344414249513 1loss/SRqqt1/detection_loss1.3221305061752597 1loss/SRqqt3/detection_loss2.5334919239244154 0loss/FRt1/assignment_loss2.0309000623987075 /loss/FRt3/assignment_loss5.110676714352199 2loss/SRqqt3/assignment_loss2.3501900011493313 1loss/FBt3/assignment_loss0.46442657424246564 /loss/FRt1/detection_loss1.7749371048300824 1loss/SRqqt4/detection_loss2.0160631402251648Mǚ^ /loss/FBt4/detection_loss2.4993760375415577 epoch0 _runtime1003.331263959 _step47 2loss/SRqqt1/assignment_loss1.0765299778897315 /loss/FRt4/detection_loss2.4041122794151306 trainer/global_step599 % _timestamp1.7884661132903872e+09 1loss/SRqqt2/assignment_loss1.846291193688238 /loss/FBt4/assignment_loss0.735863035380402 .loss/FRt3/detection_loss3.216787944034654 0loss/FRt2/assignment_loss3.6530197092464993 1loss/SRqqt2/detection_loss1.9287062718927324 /loss/FBt1/detection_loss0.7502651989704221 %loss/total_loss51.07173331686685 /loss/FBt2/detection_loss3.9267451550279344 /loss/FBt3/detection_loss3.2944064490935383 2loss/SRqqt4/assignment_loss2.0521671721308503 1loss/FBt1/assignment_loss0.22612344414249513 1loss/SRqqt1/detection_loss1.3221305061752597 1loss/SRqqt3/detection_loss2.5334919239244154 0loss/FRt1/assignment_loss2.0309000623987075 /loss/FRt3/assignment_loss5.110676714352199 2loss/SRqqt3/assignment_loss2.3501900011493313 1loss/FBt3/assignment_loss0.46442657424246564 /loss/FRt1/detection_loss1.7749371048300824 1loss/SRqqt4/detection_loss2.0160631402251648 /loss/FRt4/assignment_loss3.666990719114741 1loss/FBt2/assignment_loss0.10045322584388876 .loss/FRt2/detection_loss2.091079476156405/|Sޓ(: § gpu.0.gpu86.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp43.0 gpu.0.powerWatts118.758' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.7516 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0vu(: §% disk./.usageGB1143.6333465576172 disk.drbd1260.out 0.13671875! disk.nvme0n1.in 2332.56640625# disk.dm-0.out17677.66552734375 disk.sda3.in0 disk.sda3.out 30.72265625$ memory_percent72.42333889007568& disk.nvme0n1.out17677.66552734375 disk.sda2.out0) proc.memory.availableMB 4518.16015625 proc.cpu.threads17 disk.dm-1.in0 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0( disk.nvme0c0n1.out17677.66552734375" proc.memory.rssMB 3019.8046875* proc.memory.percent18.431425094604492 cpu24.160611846255822 network.sent945015 network.recv368401 disk.dm-1.out 30.72265625 disk.drbd1260.in0 disk.dm-0.in 2332.56640625 disk.sda.in0 disk.sda.out 30.72265625# disk.nvme0c0n1.in 2332.56640625) disk./.usagePercent76.75019345977347`s(: ѧ network.sent983781) disk./.usagePercent76.76950948423944 disk.sda1.in0 disk.sda3.out 31.07421875) proc.memory.availableMB 4441.08984375 cpu24.208881667128452 disk.dm-0.in 2332.56640625# disk.nvme0c0n1.in 2332.56640625 disk.drbd1260.in0& disk.nvme0n1.out17977.67529296875 disk.sda3.in0 disk.sda2.in0$ memory_percent72.89373874664307 proc.cpu.threads17 network.recv374176 disk.dm-1.in0# disk.dm-0.out17977.67529296875 disk.sda.in0 disk.sda2.out0# proc.memory.rssMB 3096.03515625% disk./.usageGB1143.9211692810059( disk.nvme0c0n1.out17977.67529296875 disk.dm-1.out 31.07421875 disk.drbd1260.out0.140625! disk.nvme0n1.in 2332.56640625 disk.sda1.out0 disk.sda.out 31.07421875) proc.memory.percent18.89669895172119(: ѧ gpu.0.gpu97.0 gpu.0.memory22+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts320.647' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.1294 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0B(: ͝ gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp38.0 gpu.0.powerWatts114.461' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent22.8922 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Zg(:  disk.sda2.in0 disk.dm-1.out31.5625 proc.cpu.threads17% disk./.usageGB1144.1473808288574 disk.sda2.out0( disk.nvme0c0n1.out18156.24951171875 disk.dm-1.in0 disk.drbd1260.out0.140625) proc.memory.availableMB 4422.58984375* proc.memory.percent19.008946418762207 cpu24.283812644257075( disk./.usagePercent76.7846907310199 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda3.out31.5625 disk.sda.out31.5625 disk.drbd1260.in0& disk.nvme0n1.out18156.24951171875 network.sent987483 network.recv378086 disk.dm-0.in 2332.56640625# disk.dm-0.out18156.24951171875# disk.nvme0c0n1.in 2332.56640625! disk.nvme0n1.in 2332.56640625$ memory_percent73.00665378570557# proc.memory.rssMB 3114.42578125 disk.sda.in0yq(: ˈ disk.dm-1.out 31.90234375% disk.nvme0n1.out18494.4560546875* proc.memory.percent19.005846977233887 proc.cpu.threads17% disk./.usageGB1144.1708526611328# disk.nvme0c0n1.in 2464.21484375' disk.nvme0c0n1.out18494.4560546875 disk.drbd1260.in0 disk.drbd1260.out 0.14453125 disk.sda3.out 31.90234375 cpu24.347042367031243 network.sent993296 disk.sda.in0 disk.sda2.in0! disk.nvme0n1.in 2464.21484375 disk.dm-0.in 2464.21484375" disk.dm-0.out18494.4560546875 disk.sda1.in0 disk.sda1.out0 network.recv382189 disk.sda.out 31.90234375 disk.sda2.out0 disk.dm-1.in0 disk.sda3.in0$ memory_percent73.00424575805664' proc.memory.availableMB 4422.984375# proc.memory.rssMB 3113.91796875) disk./.usagePercent76.78626594537808@똛(:  gpu.0.gpu2.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp37.0 gpu.0.powerWatts112.34' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent22.468000000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Vf(:  gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp38.0 gpu.0.powerWatts112.742' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent22.5484 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0#i(: ) proc.memory.availableMB 4422.96484375" proc.memory.rssMB 3113.9765625$ disk./.usageGB1144.175609588623# disk.nvme0c0n1.in 2464.21484375 disk.drbd1260.in0! disk.nvme0n1.in 2464.21484375 disk.dm-0.in 2464.21484375" disk.dm-0.out18594.6787109375 disk.sda1.out0 cpu24.42338429429277) disk./.usagePercent76.78658518677327 disk.dm-1.in0 disk.dm-1.out 32.390625 disk.sda3.in0 disk.sda3.out 32.390625 disk.sda2.in0$ memory_percent73.00436496734619 proc.cpu.threads17% disk.nvme0n1.out18594.6787109375 disk.sda1.in0 disk.sda.in0 disk.sda.out 32.390625) proc.memory.percent19.00620460510254 network.sent998285 network.recv385805' disk.nvme0c0n1.out18594.6787109375 disk.drbd1260.out 0.14453125 disk.sda2.out06JbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPWJbE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP;,NbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP3lNbI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPڵHbC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPʪIHbC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPN=JbE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPfC<b7 particle/specificity"trainer/global_step2:JP7NJbE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPIg+@b; particle/accuracy_1_of_7"trainer/global_step2:JPOJbE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPkoNbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPթHbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPYE!NbI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP; dHbC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPd}NbI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP'JbE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP4bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/gloYbal_step2:JP%JbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPHbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPɓJbE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP@aJbE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP=OJbE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP{ĄJbE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPgHbC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPcoJbE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPJHbC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPKHbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP*ɠJbE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP_HbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPc}HbC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP]HbC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPwHbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP:VJbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP!NbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP7WJbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP*NbI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP[NbI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPJC;b6 jet/accuracy_3_of_4"trainer/global_step2:JPOKJbE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPHbC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPQtmzHbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPy-lJbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP7JbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPINbI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPrJbE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPH;b6 jet/accuracy_1_of_7"trainer/global_step2:JP>JbE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP5JbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP $NbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPU7JbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP+HbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP`)5NbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP/WHbC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP=VJbE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP(itNbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÏJbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPv NbI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP ݺ@b; particle/accuracy_2_of_3"trainer/global_step2:JP'HbC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP6A0JbE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPB'HbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPdJbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP@&JbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP1je'JbE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP>ZKJbE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP^e NbI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP1JbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP~;b6 jet/accuracy_3_of_5"trainer/global_step2:JPAXƓHbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPJbE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPodHbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP"V@b; particle/accuracy_4_of_5"trainer/global_step2:JPޮwJbE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP~DhNbI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP`JbE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPGoNbI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPhNbI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP5hNbI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP%NbI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP韃BJbE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPP>HbC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPQNbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPNbI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPHĿNbI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP?JbE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP16HHbC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPʆJbE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPJbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP.HbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPJbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJܓNbI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPNbI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP?^l;b6 jet/accuracy_1_of_3"trainer/global_step2:JPA=NbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPGaJbE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPS2NbI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPa}NbI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPe-@b; particle/accuracy_1_of_3"trainer/global_step2:JPb GJbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP8HbC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP\nNbI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPDHbC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPʉ3:JbE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPa~+JbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPAHbC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPNbI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPi['NbI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP7_HbC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPGJbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP'";b6 jet/accuracy_5_of_7"trainer/global_step2:JPJbE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP-JbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPn[JbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPIZNbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPz HbC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPfJbE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP!JbE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPv HbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPCsm;b6 jet/accuracy_4_of_6"trainer/global_step2:JPHbC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPoWNbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP (JbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP!JbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPfHbC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPiNbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP/ HbC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP %JbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPOHbC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPM7JbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP|"JbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPAHbC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPOHbC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPpjSHbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP HbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP|JbE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP\=3HbC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP;_HbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPHbC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPNbI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPLB JbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPajAJbE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPٛNbI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPQJbE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP9>]JbE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPחNbI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP؋HbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP@IJbE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP HbC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPJbE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP[uHbC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP"'JbE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPypJbE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP|ߣNbI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPqkNbI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPvS!HbC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPL;LJbE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPcɻ9NbI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPJbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP:NbI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP!HbC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPYJbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP, HbC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP]NbI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPm=/JbE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPq-TYNbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPv0JbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPaLNbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPJbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPKHbC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPFHbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPFJbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPҍNbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPyRJbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP_%NbI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP]@b; particle/accuracy_3_of_3"trainer/global_step2:JPJbE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPHbC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPjgYHbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP(NbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP&HbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPJbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPJbE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPeLNbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP&"NbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP ymJbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP,DJbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP$PHbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPi+HbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPd|JbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPk ;b6 jet/accuracy_3_of_3"trainer/global_step2:JP֟=JbE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPK>HbC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP+鹏HbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPDRNbI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPTNbI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP.HbC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPnHbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPoJbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJbE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPP@b; particle/accuracy_2_of_6"trainer/global_step2:JPMNbI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP)SRHbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP NbI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPA;b6 jet/accuracy_3_of_7"trainer/global_step2:JP c|HbC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPNbI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPa HbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPpJbE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPPCHbC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP,JbE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPhDNbI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPWGbB validation_average_jet_accuracy"trainer/global_step2:JP"w0HbC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP v\HbC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPstHbC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPBvHbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPhNbI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP(JbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP:]JbE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPcny6HbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP2NbI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP(:0NbI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPljHbC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPf@b; particle/accuracy_3_of_4"trainer/global_step2:JP};b6 jet/accuracy_1_of_6"trainer/global_step2:JP^JbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPo HbC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPY}HbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP:oJbE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPjGHbC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP^pHbC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPJbE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-|HaHbC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPf[JbE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPRBJbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPr3HbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPwۥRJbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPmMm NbI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP HbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPMJbE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPVm@b; particle/accuracy_4_of_6"trainer/global_step2:JP}QJbE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPRJbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP{bG;b6 jet/accuracy_1_of_2"trainer/global_step2:JPݥNbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP.=HbC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPt( JbE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPwHbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPYHbC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPHbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPNbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPAJbE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!EeHbC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP6hJbE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP^JbE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPތzHbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP'weNbI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPSgJbE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP~|JbE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPkNbI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPTJbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP~^'JbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP<HbC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPᣔNbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP JbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPHbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP{NbI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP_JbE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP~UHbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPN)@b; particle/accuracy_5_of_7"trainer/global_step2:JPJbE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPIQcJbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP4JbE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP7'vNbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP%WNbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP}mHbC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPf2HbC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPmJbE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP<NbI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPR) HNbI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPĺBNbI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP JbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPƜHbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPydHbC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPy2JbE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP2WNbI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP&`NbI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPu&;b6 jet/accuracy_5_of_6"trainer/global_step2:JP|a9HbC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPwHbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPAZNbI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP!JbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPANbI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPJbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP(BYJbE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPXe(HbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPs55NbI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP9JbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP^NbI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP=V K@b; particle/accuracy_4_of_7"trainer/global_step2:JPesNbI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPNbI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP1ޖNbI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP"NbI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP P{;b6 jet/accuracy_2_of_4"trainer/global_step2:JP]RNbI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPJbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPG i@b; particle/accuracy_2_of_2"trainer/global_step2:JPJbE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP-HbC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP͠JbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPUhHbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPOЅJbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP0P&HbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPl";b6 jet/accuracy_1_of_5"trainer/global_step2:JP(HbC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPv~JbE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPAHbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPNbI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPx18JbE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPu0@b; particle/accuracy_1_of_2"trainer/global_step2:JPBnXJbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP[NbI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP>-yNbI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPKFNbI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP3$NbI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPgD;b6 jet/accuracy_5_of_5"trainer/global_step2:JP E@b; particle/accuracy_1_of_1"trainer/global_step2:JPpBKJbE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPBlNbI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPcJbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPXGJ;b6 jet/accuracy_1_of_1"trainer/global_step2:JPz-tHbC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP0JbE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPrnJbE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPACgJbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPxJbE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP8JbE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP{hXJbE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPIlJbE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP0NbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP8QXHbC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP,JbE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPܣJbE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP%\4HbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP[[JbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPEJbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPHcJbE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP^NbI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPQJbE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP`CHbC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPͰnHbC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP}%NbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP69JbE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPgONbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPkMJbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP NbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPR/JbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP5z@b; particle/accuracy_2_of_5"trainer/global_step2:JPy;;b6 jet/accuracy_6_of_6"trainer/global_step2:JPeHbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPVJbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP1=NbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPHbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPl{HbC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÈo@b; particle/accuracy_4_of_4"trainer/global_step2:JP XHbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPĖeNbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP;25JbE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPxƦHbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP?DJbE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7pNbI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPR;b6 jet/accuracy_7_of_7"trainer/global_step2:JPy%HbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP~uNbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP<8JbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJ#HbC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP2JbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP*NbI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP: hHbC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPNbI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPPNbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP<ѣJbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPunHbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPI1NNbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP"sHbC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP6u<b7 particle/sensitivity"trainer/global_step2:JP$ʧJbE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPplJbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP_^JbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPNbI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPsJbE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP)?lNbI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPdedHbC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPž4NbI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPPHbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPNbI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPM?JbE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP1HbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPyJbE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPM5NbI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP+a;b6 jet/accuracy_2_of_7"trainer/global_step2:JPaHbC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPzQeGHbC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP8JbE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPpxHbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPy,NbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPM*HbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP+NbI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPdOHbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP2?N@b; particle/accuracy_3_of_5"trainer/global_step2:JP3T@b; particle/accuracy_6_of_7"trainer/global_step2:JPHbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP"HbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP3wHbC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP;JbE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPe)NbI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP4HbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP}JbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP96vJbE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP.NbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPOJbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPlJbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPbJbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPwsJbE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP])!HbC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPҷNbI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP ĐJbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPHbC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPGNbI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP ;b6 jet/accuracy_2_of_2"trainer/global_step2:JP5PJbE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPJbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP gJbE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP<<yNbI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP1:2;b6 jet/accuracy_1_of_4"trainer/global_step2:JPIgb;b6 jet/accuracy_2_of_5"trainer/global_step2:JP|J(HbC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPHbC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPfHbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPf_8HbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPjJbE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPNbI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPxHbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPTJbE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPȲJbE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP&1>NbI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPkȂJbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPŰHbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPHHbC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPj9LJbE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP@*CJbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPQHbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPJJbE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP^NtNbI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPg&qJbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPbNbI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPV @UNbI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP$'JbE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP'HbC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPs9+ HbC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP$pHbC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPfHbC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPeBHbC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPBʇ;b6 jet/accuracy_3_of_6"trainer/global_step2:JPQ5HbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPuϊwNbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPc9NbI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP?6NbI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPHbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPHJbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPfNbI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP,HbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP:JbE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP7|HbC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPpy>JbE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP 7JbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPNgJbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP,ڤNbI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPǩ6NbI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP3CHbC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP0;b6 loss/val_total_loss"trainer/global_step2:JPZ JbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPMkNbI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPlNbI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP*ԁ@b; particle/accuracy_5_of_5"trainer/global_step2:JPwJbE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP#cNbI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPd=HbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP5)NbI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPqpNbI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPJbE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPUåHbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP<kNbI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPɄYNbI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPHbC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPHbC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPnQ@b; particle/accuracy_2_of_7"trainer/global_step2:JP.ZHbC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP֢aNbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP@+mHbC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPpFJbE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP`JbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP*HbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPG[JbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP?P7NbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPLJbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPFJbE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPREJbE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7HbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPN bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP J bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPLJ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPdXdN bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP2;(J bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPM@ b; particle/accuracy_1_of_5"trainer/global_step2:JPJ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-@ b; particle/accuracy_3_of_7"trainer/global_step2:JPQ5TtJ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPN bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP:N bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPaJ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPPyH bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP+9N bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP\U%KH bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP=[p N bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP\8(eJ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPNՔJ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP7}J bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPH bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP'GN bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPWN bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP{N bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPn+iH bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPMN bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPN bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPklN bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP׿N bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPEN bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPWN bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP&nN bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPiJ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP-= J bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP:$H bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP!N bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPR N bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPkN bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP1N bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPZN bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPBN bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP .}N bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP pJ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP`H bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP۩qJ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP69 9N bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP .J bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP?J bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPg:H bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPzrN bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPbN bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPGH bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPocJ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPJ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP (J bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP2N bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPRzH bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPY|H bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP J bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\l~@ b; particle/accuracy_7_of_7"trainer/global_step2:JPsBJ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPUJ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPQ*H bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPF.N bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP4AN bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPK|P9 b4 particle/accuracy"trainer/global_step2:JPHR N bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPyަN bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP\5fN bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPlبJ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPJON bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPHN bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP ~O J bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPXm N bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPeN bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP(}H bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPu,@ b; particle/accuracy_3_of_6"trainer/global_step2:JPm; b6 jet/accuracy_6_of_7"trainer/global_step2:JPH bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPOdJ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPζJ bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPve*N bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPYN bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPvj J bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP[N bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPJ!); b6 jet/accuracy_2_of_3"trainer/global_step2:JP}D[H bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP=oH bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP~,H bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP~J bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP,d}N bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP9; b6 jet/accuracy_4_of_5"trainer/global_step2:JPߎJ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPğ*H bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP1H bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPfH bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPLJ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP H bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPbJ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP/N bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPN bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPp")H bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP,\H bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP%fN bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP@ b; particle/accuracy_5_of_6"trainer/global_step2:JP!; b6 jet/accuracy_4_of_7"trainer/global_step2:JPFiH bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP{n2J bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPJ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPfN bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPqXYJ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPi J bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPTH bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPEHH bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPgN bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPE?J bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPN bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPaNgJ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP:H bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPJ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP* _N bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPGR5N bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPkH bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP]oH bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPP N bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPY~N bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP_H bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP_H bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPXN bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP pJ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP!iN bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPN bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPDSN bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP|J bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP'K@ b; particle/accuracy_1_of_6"trainer/global_step2:JP!vJ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP J bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP韽N bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPH bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP N bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP'+N bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP\J bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPl?*H bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPJ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPnN bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPN bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP{N bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPN bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP(N bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPϓN bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPtN bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPafJ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPLBN bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP[J bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP(OH bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP+2>J bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPH bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP12 - _wandbruntime1065  _runtime1065?BU#  ("Purity/1FRt3SRqqt*FBt/event_purity0 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.34431639313697815 ("Purity/3FRt0SRqqt0FBt/event_purity0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.38271602988243103 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ;"Purity/1FRt1SRqqt*FBt/event_purity0.010729643516242504 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.16644905507564545 & Purity/4FRt1SRqqt*FBt/FRt_purity0 ("Purity/4FRt1SRqqt0FBt/event_purity0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.3075000047683716 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9927419424057007 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.5 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 ("Purity/1FRt2SRqqt1FBt/event_purity0 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 ("Purity/2FRt4SRqqt0FBt/event_purity0 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.3055555522441864 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8868313431739807 8 Purity/3FRt*SRqqt0FBt/FRt_purity0.13566139340400696 & Purity/3FRt0SRqqt*FBt/FBt_purity1 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.1666666716337204 +jet/accuracy_3_of_60.07017543911933899 trainer/global_step603 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.21987119317054749 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 6 Purity/*FRt1SRqqt1FBt/FBt_purity0.994231641292572 ;"Purity/1FRt1SRqqt0FBt/event_purity0.012278445065021515 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity0.20726962387561798 ("Purity/0FRt3SRqqt0FBt/event_purity0 & Purity/1FRt0SRqqt4FBt/FRt_purity0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity0 9"Purity/*FRt*SRqqt0FBt/event_purity0.1390431523323059 :"Purity/*FRt1SRqqt*FBt/event_purity0.15690869092941284 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.0476190485060215 )loss/val_total_loss49.97751034713482 ;"Purity/*FRt1SRqqt1FBt/event_purity0.054934509098529816 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 particle/accuracy_5_of_51 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9787802696228027 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.375 _runtime1065.341910754 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9951668977737427 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity1 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.12745098769664764 particle/accuracy_2_of_71 & Purity/0FRt3SRqqt*FBt/FBt_purity1 % _timestamp1.7884661752978544e+09 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.3205128014087677 ("Purity/1FRt*SRqqt4FBt/event_purity0 9"Purity/0FRt*SRqqt2FBt/event_purity0.2613388001918793 8 Purity/2FRt0SRqqt0FBt/FRt_purity0.14929743111133575 :"Purity/2FRt0SRqqt2FBt/event_purity0.02500000037252903 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ("Purity/2FRt1SRqqt1FBt/event_purity0 ("Purity/2FRt3SRqqt0FBt/event_purity0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.3368164896965027 7 Purity/1FRt1SRqqt2FBt/FRt_purity0.4009353816509247 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.6190475821495056 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.4125337302684784 particle/accuracy_1_of_51 ."Purity/*FRt4SRqqt0FBt/SRqqt_purity0.34375 particle/accuracy_3_of_71 ("Purity/4FRt1SRqqt*FBt/event_purity0 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.41132956743240356 8 Purity/*FRt1SRqqt2FBt/FRt_purity0.35067036747932434 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 8 Purity/3FRt1SRqqt*FBt/FRt_purity0.18930043280124664 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ("Purity/*FRt3SRqqt*FBt/event_purity0 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9967741966247559 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/1FRt0SRqqt4FBt/FBt_purity1 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.3859025239944458 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9842396974563599 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 & Purity/3FRt*SRqqt2FBt/FBt_purity1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ("Purity/1FRt4SRqqt*FBt/event_purity0 _step48 9"Purity/0FRt*SRqqt3FBt/event_purity0.1358024626970291 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9953917264938354 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.4285714328289032 <"Purity/1FRt1SRqqt2FBt/event_purity0.0071428571827709675 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 8 Purity/2FRt*SRqqt2FBt/FRt_purity0.39000001549720764 & Purity/3FRt1SRqqt1FBt/FBt_purity1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.3382842242717743 particle/accuracy_7_of_70.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity0 ("Purity/*FRt1SRqqt3FBt/event_purity0 8 Purity/*FRt0SRqqt2FBt/FRt_purity0.40742093324661255 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 (particle/accuracy0.8352339267730713 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 :"Purity/1FRt0SRqqt0FBt/event_purity0.14258244633674622 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.588888943195343 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.17743654549121857 particle/accuracy_3_of_61 jet/accuracy_6_of_70 & Purity/1FRt4SRqqt0FBt/FRt_purity0 ("Purity/0FRt1SRqqt3FBt/event_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 ;"Purity/2FRt*SRqqt2FBt/event_purity0.017777778208255768 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 +jet/accuracy_2_of_30.18297651410102844 8 Purity/2FRt4SRqqt*FBt/FRt_purity0.30000001192092896 & Purity/2FRt2SRqqt2FBt/FBt_purity1 & Purity/4FRt0SRqqt*FBt/FRt_purity0 :"Purity/*FRt0SRqqt2FBt/event_purity0.26113954186439514 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 jet/accuracy_4_of_50 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity0 & Purity/1FRt3SRqqt2FBt/FRt_purity1 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8837166428565979 & Purity/*FRt3SRqqt*FBt/FBt_purity1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.35856306552886963 7 Purity/1FRt*SRqqt2FBt/FRt_purity0.4352588653564453 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.32949307560920715 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 7 Purity/*FRt2SRqqt2FBt/FRt_purity0.32051283121109/?wEp01 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32400012016296387 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 particle/accuracy_5_of_61 jet/accuracy_4_of_70 & Purity/2FRt0SRqqt3FBt/FRt_purity1 :"Purity/*FRt0SRqqt0FBt/event_purity0.12108093500137329 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.37786826491355896 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.4000000059604645 ("Purity/0FRt4SRqqt0FBt/event_purity0 & Purity/2FRt3SRqqt*FBt/FBt_purity1 & Purity/*FRt3SRqqt2FBt/FBt_purity1 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 9"Purity/0FRt0SRqqt1FBt/event_purity0.9979397654533386 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt*SRqqt3FBt/event_purity0 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.21315227448940277 :"Purity/0FRt2SRqqt*FBt/event_purity0.04218094423413277 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 7 Purity/2FRt2SRqqt*FBt/FBt_purity0.9821428656578064 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.44736841320991516 & Purity/0FRt4SRqqt*FBt/FBt_purity1 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 :"Purity/*FRt2SRqqt0FBt/event_purity0.01924903318285942 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 ("Purity/0FRt3SRqqt*FBt/event_purity0 particle/accuracy_1_of_61 ("Purity/*FRt1SRqqt4FBt/event_purity0 9"Purity/0FRt0SRqqt2FBt/event_purity0.3327980935573578 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.14258244633674622 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 :"Purity/*FRt*SRqqt2FBt/event_purity0.19419775903224945 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.24562212824821472 ;"Purity/2FRt*SRqqt0FBt/event_purity0.003246735082939267 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.34561991691589355 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 ;"Purity/*FRt2SRqqt1FBt/event_purity0.004378923214972019 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9925950169563293 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4279800355434418 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9947412610054016 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.3674633204936981 ("Purity/3FRt0SRqqt*FBt/event_purity0 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.1388888955116272 7 Purity/0FRt1SRqqt3FBt/FBt_purity0.8666666150093079 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.40637654066085815 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 +particle/specificity0.9506799578666687 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.3499999940395355 particle/accuracy_1_of_71 ("Purity/*FRt2SRqqt3FBt/event_purity0 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9836632013320923 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ("Purity/4FRt2SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt1FBt/FRt_purity0.2983627915382385 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.3935238718986511 6 Purity/2FRt2SRqqt0FBt/FRt_purity0.151878222823143 ("Purity/0FRt*SRqqt4FBt/event_purity0 7"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.69047611951828 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity0 ("Purity/3FRt*SRqqt0FBt/event_purity0 + Purity/0FRt0SRqqt4FBt/FBt_purity0.8125 ("Purity/3FRt4SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.9166666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.22454558312892914 ."Purity/1FRt*SRqqt3FBt/event_purity0.03125 7 Purity/2FRt*SRqqt1FBt/FBt_purity0.9910394549369812 8 Purity/2FRt3SRqqt0FBt/FRt_purity0.21369047462940216 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9962766766548157 :"Purity/0FRt*SRqqt*FBt/event_purity0.40748661756515503 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 :"Purity/1FRt*SRqqt0FBt/event_purity0.06467058509588242 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 +jet/accuracy_3_of_40.02782217226922512 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.20000000298023224 7 Purity/*FRt3SRqqt0FBt/FRt_purity0.1729462444782257 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9952755570411682 :"Purity/1FRt*SRqqt2FBt/event_purity0.10495587438344955 ("Purity/2FRt1SRqqt*FBt/event_purity0 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 ("Purity/3FRt2SRqqt0FBt/event_purity0 jet/accuracy_1_of_71 ("Purity/1FRt4SRqqt0FBt/event_purity0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.3713327944278717 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.35863494873046875 7 Purity/1FRt1SRqqt*FBt/FRt_purity0.2292298525571823 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 & Purity/3FRt2SRqqt1FBt/FBt_purity1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity0.25 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 :"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.43421053886413574 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 particle/accuracy_2_of_31 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8968254327774048 ("Purity/3FRt3SRqqt*FBt/event_purity0 8 Purity/*FRt1SRqqt1FBt/FRt_purity0.31259778141975403 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.3663094937801361 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.4014635682106018 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.6190475821495056 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 ("Purity/*FRt3SRqqt1FBt/event_purity0 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.39810892939567566 +jet/accuracy_3_of_50.06857157498598099 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9798036217689514 ("Purity/1FRt3SRqqt1FBt/event_purity0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 particle/accuracy_4_of_51 ("Purity/1FRt1SRqqt4FBt/event_purity0 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 ("Purity/3FRt*SRqqt*FBt/event_purity0 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 ("Purity/2FRt4SRqqt*FBt/event_purity0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2916666567325592 & Purity/4FRt*SRqqt*FBt/FRt_purity0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 9"Purity/1FRt0SRqqt1FBt/event_purity0.1138874739408493 8 Purity/1FRt0SRqqt2FBt/FRt_purity0.44340893626213074 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.3943653702735901 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 *jet/accuracy_1_of_30.8152989149093628 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 ("Purity/2FRt2SRqqt1FBt/event_purity0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 particle/accuracy_1_of_31 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.38908833265304565 8 Purity/3FRt*SRqqt1FBt/FRt_purity0.30909091234207153 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.13405796885490417 ("Purity/0FRt2SRqqt2FBt/event_purity0 9"Purity/*FRt0SRqqt1FBt/event_purity0.6299179196357727 & Purity/*FRt3SRqqt1FBt/FBt_purity1 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 ;"Purity/1FRt1SRqqt1FBt/event_purity0.005673849955201149 jet/accuracy_5_of_70 ("Purity/2FRt2SRqqt2FBt/event_purity0 ;"Purity/*FRt2SRqqt*FBt/event_purity0.016647348180413246 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.40642601251602173 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 & Purity/3FRt1SRqqt*FBt/FBt_purity1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.2587719261646271 ("Purity/*FRt3SRqqt2FBt/event_purity0 8 Purity/2FRt1SRqqt0FBt/FRt_purity0.17026859521865845 jet/accuracy_4_of_60 ( Purity/*FRt1SRqqt3FBt/FRt_purity0.5 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 :"Purity/1FRt0SRqqt2FBt/event_purity0.15711678564548492 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.3710061311721802 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.15687404572963715 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 & Purity/4FRt1SRqqt0FBt/FRt_purity0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.37181708216667175 ( Purity/1FRt1SRqqt3FBt/FRt_purity0.5 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.36909013986587524 ("Purity/2FRt1SRqqt2FBt/event_purity0 & Purity/*FRt*SRqqt4FBt/FRt_purity0 & Purity/4FRt2SRqqt0FBt/FRt_purity0 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.15067921578884125 7 Purity/1FRt*SRqqt0FBt/FRt_purity0.1686033457517624 ("Purity/3FRt4SRqqt0FBt/event_purity0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt0FBt/FRt_purity0.1462327092885971 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.8999999761581421 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ;"Purity/2FRt*SRqqt*FBt/event_purity0.005416110623627901 ("Purity/3FRt1SRqqt1FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.5 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 7 Purity/2FRt*SRqqt0FBt/FRt_purity0.1641518771648407 ("Purity/*FRt4SRqqt0FBt/event_purity0 & Purity/2FRt3SRqqt1FBt/FBt_purity1 ("Purity/2FRt0SRqqt3FBt/event_purity0 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 ("Purity/3FRt3SRqqt0FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 & Purity/0FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt0SRqqt2FBt/event_purity0 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity0.027037035673856735 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 7 Purity/2FRt2SRqqt1FBt/FBt_purity0.9821428656578064 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.3586941063404083 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.18539854884147644 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ("Purity/3FRt2SRqqt1FBt/event_purity0 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.2977248728275299 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.4177035391330719 8 Purity/3FRt0SRqqt*FBt/FRt_purity0.23232322931289673 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9845708012580872 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.37401270866394043 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ("Purity/1FRt2SRqqt*FBt/event_purity0 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 particle/accuracy_3_of_31 ("Purity/3FRt*SRqqt2FBt/event_purity0 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.0476190485060215 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9734309911727905 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9882583618164062 ;"Purity/2FRt*SRqqt1FBt/event_purity0.010880696587264538 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity0 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.43421053886413574 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.42955464124679565 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9851446747779846 6 Purity/1FRt*SRqqt*FBt/FBt_purity0.985005259513855 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.3771679699420929 +jet/accuracy_3_of_30.01692398637533188 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.34414681792259216 & Purity/*FRt3SRqqt2FBt/FRt_purity1 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9941508173942566 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 & Purity/1FRt3SRqqt*FBt/FBt_purity1 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.410755455493927 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.3499999940395355 particle/accuracy_2_of_61 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.20129871368408203 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 jet/accuracy_3_of_70 & Purity/*FRt1SRqqt4FBt/FRt_purity0 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ("Purity/0FRt4SRqqt1FBt/event_purity0 & Purity/1FRt3SRqqt2FBt/FBt_purity1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.33078595995903015 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 7validation_average_jet_accuracy0.38241782784461975 & Purity/1FRt1SRqqt4FBt/FRt_purity0 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.47058823704719543 7 Purity/2FRt0SRqqt1FBt/FRt_purity0.2680951952934265 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 ("Purity/2FRt2SRqqt*FBt/event_purity0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity0 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9732359647750854 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.11388888210058212 particle/accuracy_3_of_41 *jet/accuracy_1_of_60.9210526347160339 ("Purity/*FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9605262875556946 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9796597361564636 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.3400500416755676 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 8 Purity/3FRt0SRqqt1FBt/FRt_purity0.37179484963417053 & Purity/3FRt2SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.3076923191547394 & Purity/2FRt0SRqqt3FBt/FBt_purity1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 :"Purity/*FRt*SRqqt1FBt/event_purity0.35935112833976746 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9830425381660461 9"Purity/0FRt*SRqqt0FBt/event_purity0.2963734269142151 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9911110997200012 ("Purity/*FRt4SRqqt1FBt/event_purity0 particle/accuracy_4_of_61 ("Purity/4FRt*SRqqt0FBt/event_purity0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.36995217204093933 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 *jet/accuracy_1_of_20.6955632567405701 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.8333333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity0 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9805105328559875 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.15480343997478485 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 /"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.328125 & Purity/1FRt4SRqqt*FBt/FRt_purity0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt0FBt/event_purity0 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.36147934198379517 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.40942031145095825 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.375 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 9"Purity/0FRt1SRqqt0FBt/event_purity0.3713327944278717 ;"Purity/0FRt2SRqqt0FBt/event_purity0.049589913338422775 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.8571428656578064 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.3768426477909088 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9606753587722778 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 ("Purity/3FRt2SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9871768951416016 particle/accuracy_5_of_71 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 ("Purity/2FRt2SRqqt0FBt/event_purity0 ("Purity/3FRt*SRqqt1FBt/event_purity0 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 8 Purity/2FRt4SRqqt0FBt/FRt_purity0.30000001192092896 & Purity/4FRt0SRqqt0FBt/FRt_purity0 :"Purity/*FRt*SRqqt3FBt/event_purity0.08908046036958694 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 :"Purity/*FRt*SRqqt*FBt/event_purity0.20481795072555542 8 Purity/*FRt*SRqqt0FBt/FRt_purity0.16521237790584564 & Purity/3FRt0SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.4305555522441864 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_5_of_60 & Purity/1FRt*SRqqt4FBt/FRt_purity0 7 Purity/*FRt1SRqqt*FBt/FBt_purity0.9841686487197876 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 ("Purity/1FRt2SRqqt0FBt/event_purity0 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 +"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.25 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9690475463867188 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 ;"Purity/2FRt0SRqqt*FBt/event_purity0.014877114444971085 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 particle/accuracy_4_of_71 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 *jet/accuracy_2_of_40.2762931287288666 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.3700048625469208 particle/accuracy_2_of_21 ("Purity/*FRt4SRqqt*FBt/event_purity0 & Purity/*FRt4SRqqt*FBt/FBt_purity1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.3565407991409302 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9574074149131775 :"Purity/2FRt0SRqqt1FBt/event_purity0.01944444514811039 7 Purity/2FRt0SRqqt1FBt/FBt_purity0.9916666746139526 )jet/accuracy_1_of_50.918146550655365 & Purity/*FRt0SRqqt4FBt/FRt_purity0 ("Purity/0FRt0SRqqt0FBt/event_purity1 5 Purity/1FRt0SRqqt*FBt/FRt_purity0.22305928170681 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.36798471212387085 particle/accuracy_1_of_21 ;"Purity/*FRt1SRqqt2FBt/event_purity0.020737329497933388 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 jet/accuracy_5_of_50 particle/accuracy_1_of_11 *"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.37597399950027466 *jet/accuracy_1_of_10.4200733006000519 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9583333134651184 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 9"Purity/0FRt0SRqqt3FBt/event_purity0.1733333170413971 ("Purity/2FRt1SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt0FBt/event_purity0 ("Purity/0FRt4SRqqt*FBt/event_purity0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.2857142984867096 ("Purity/4FRt0SRqqt*FBt/event_purity0 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.1388888955116272 ("Purity/2FRt3SRqqt1FBt/event_purity0 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.18958131968975067 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.3994259238243103 ;"Purity/1FRt*SRqqt1FBt/event_purity0.058573395013809204 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.38349446654319763 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt3SRqqt*FBt/event_purity0 8 Purity/2FRt3SRqqt*FBt/FRt_purity0.18909522891044617 + Purity/0FRt*SRqqt4FBt/FBt_purity0.8125 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 9"Purity/*FRt0SRqqt3FBt/event_purity0.1011904701590538 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 :"Purity/1FRt*SRqqt*FBt/event_purity0.06555938720703125 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 ;"Purity/2FRt0SRqqt0FBt/event_purity0.010406260378658772 particle/accuracy_2_of_51 jet/accuracy_6_of_60 6 Purity/*FRt*SRqqt*FBt/FBt_purity0.984567403793335 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.36806002259254456 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.17888912558555603 & Purity/3FRt*SRqqt1FBt/FBt_purity1 particle/accuracy_4_of_41 & Purity/0FRt3SRqqt1FBt/FBt_purity1 8 Purity/*FRt3SRqqt1FBt/FRt_purity0.22777779400348663 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 9"Purity/1FRt0SRqqt3FBt/event_purity0.0357142873108387 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 particle/accuracy_2_of_41 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9746752381324768 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.25264546275138855 7 Purity/2FRt0SRqqt2FBt/FRt_purity0.3738095462322235 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.3350158929824829 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 jet/accuracy_7_of_70 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9979397654533386 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.3820827603340149 + Purity/1FRt*SRqqt3FBt/FRt_purity0.4375 :"Purity/1FRt0SRqqt*FBt/event_purity0.13588568568229675 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.27037036418914795 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 8 Purity/2FRt3SRqqt1FBt/FRt_purity0.15000000596046448 9"Purity/*FRt0SRqqt*FBt/event_purity0.3486631214618683 7 Purity/*FRt3SRqqt*FBt/FRt_purity0.1674565225839615 6 Purity/0FRt2SRqqt*FBt/FBt_purity0.984639048576355 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 7 Purity/0FRt2SRqqt2FBt/FBt_purity0.9615384340286255 ("Purity/1FRt2SRqqt2FBt/event_purity0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 ("Purity/0FRt3SRqqt1FBt/event_purity0 7 Purity/1FRt1SRqqt0FBt/FRt_purity0.2018723040819168 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 8 Purity/2FRt2SRqqt1FBt/FRt_purity0.25654762983322144 particle/accuracy_1_of_41 7 Purity/*FRt0SRqqt4FBt/FBt_purity0.8500000238418579 epoch0 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 *jet/accuracy_2_of_60.4561403691768646 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity0.875 & Purity/4FRt*SRqqt0FBt/FRt_purity0 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 :"Purity/0FRt1SRqqt*FBt/event_purity0.29666027426719666 &particle/f_score0.286234974861145 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.46666666865348816 9"Purity/0FRt0SRqqt*FBt/event_purity0.8151975274085999 ,jet/accuracy_4_of_40.004463689401745796 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_6_of_61 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 :"Purity/0FRt1SRqqt1FBt/event_purity0.09893309324979782 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9918307662010193 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 & Purity/2FRt*SRqqt3FBt/FRt_purity1 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.46828174591064453 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9967661499977112 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.14467592537403107 ,particle/sensitivity0.21157196164131165 ("Purity/1FRt0SRqqt4FBt/event_purity0 :"Purity/0FRt2SRqqt1FBt/event_purity0.01075268816202879 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.33852043747901917 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 ("Purity/3FRt0SRqqt1FBt/event_purity0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.8835318684577942 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity0.3206464946269989 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ("Purity/*FRt0SRqqt4FBt/event_purity0 8 Purity/*FRt0SRqqt1FBt/FRt_purity0.34754031896591187 ("Purity/1FRt3SRqqt0FBt/event_purity0 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 jet/accuracy_2_of_70 & Purity/2FRt*SRqqt3FBt/FBt_purity1 & Purity/3FRt0SRqqt2FBt/FBt_purity1 ("Purity/4FRt*SRqqt*FBt/event_purity0 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2465856820344925 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.32545289397239685 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 8 Purity/2FRt1SRqqt*FBt/FRt_purity0.21749290823936462 particle/accuracy_3_of_51 particle/accuracy_6_of_71 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.39191603660583496 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9910756349563599 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.8750000596046448 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.3300520181655884 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 7 Purity/2FRt0SRqqt2FBt/FBt_purity0.9940475821495056 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.450892835855484 ("Purity/3FRt1SRqqt*FBt/event_purity0 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 ("Purity/*FRt3SRqqt0FBt/event_purity0 9"Purity/0FRt*SRqqt1FBt/event_purity0.6166820526123047 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.3975643217563629 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity0 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.36260172724723816 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.25555554032325745 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 +jet/accuracy_2_of_20.07067140191793442 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 :"Purity/*FRt1SRqqt0FBt/event_purity0.19537661969661713 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.5 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 *jet/accuracy_1_of_40.8650160431861877 *jet/accuracy_2_of_50.3450578451156616 & Purity/*FRt4SRqqt1FBt/FBt_purity1 & Purity/4FRt2SRqqt*FBt/FRt_purity0 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.17273478209972382 8 Purity/1FRt*SRqqt*FBt/FRt_purity0.21822147071361542}9/  7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9734309911727905 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9882583618164062 ;"Purity/2FRt*SRqqt1FBt/event_purity0.010880696587264538 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity0 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.43421053886413574 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.42955464124679565 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9851446747779846 6 Purity/1FRt*SRqqt*FBt/FBt_purity0.985005259513855 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.3771679699420929 +jet/accuracy_3_of_30.01692398637533188 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.34414681792259216 & Purity/*FRt3SRqqt2FBt/FRt_purity1 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9941508173942566 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 & Purity/1FRt3SRqqt*FBt/FBt_purity1 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.410755455493927 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.3499999940395355 particle/accuracy_2_of_61 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.20129871368408203 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 jet/accuracy_3_of_70 & Purity/*FRt1SRqqt4FBt/FRt_purity0 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ("Purity/0FRt4SRqqt1FBt/event_purity0 & Purity/1FRt3SRqqt2FBt/FBt_purity1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.33078595995903015 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 7validation_average_jet_accuracy0.38241782784461975 & Purity/1FRt1SRqqt4FBt/FRt_purity0 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.47058823704719543 7 Purity/2FRt0SRqqt1FBt/FRt_purity0.2680951952934265 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 ("Purity/2FRt2SRqqt*FBt/event_purity0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity0 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9732359647750854 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.11388888210058212 particle/accuracy_3_of_41 *jet/accuracy_1_of_60.9210526347160339 ("Purity/*FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9605262875556946 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9796597361564636 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.3400500416755676 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 8 Purity/3FRt0SRqqt1FBt/FRt_purity0.37179484963417053 & Purity/3FRt2SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.3076923191547394 & Purity/2FRt0SRqqt3FBt/FBt_purity1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 :"Purity/*FRt*SRqqt1FBt/event_purity0.35935112833976746 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9830425381660461 9"Purity/0FRt*SRqqt0FBt/event_purity0.2963734269142151 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9911110997200012 ("Purity/*FRt4SRqqt1FBt/event_purity0 particle/accuracy_4_of_61 ("Purity/4FRt*SRqqt0FBt/event_purity0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.36995217204093933 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 *jet/accuracy_1_of_20.6955632567405701 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.8333333134651184 ("Purity/4FRt0SRqqt0FBt/event_purity0 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9805105328559875 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.15480343997478485 @&PurP4$ity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 /"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.328125 & Purity/1FRt4SRqqt*FBt/FRt_purity0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt0FBt/event_purity0 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.36147934198379517 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.40942031145095825 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.375 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 9"Purity/0FRt1SRqqt0FBt/event_purity0.3713327944278717 ;"Purity/0FRt2SRqqt0FBt/event_purity0.049589913338422775 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.8571428656578064 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.3768426477909088 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9606753587722778 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 ("Purity/3FRt2SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9871768951416016 particle/accuracy_5_of_71 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 ("Purity/2FRt2SRqqt0FBt/event_purity0 ("Purity/3FRt*SRqqt1FBt/event_purity0 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 8 Purity/2FRt4SRqqt0FBt/FRt_purity0.30000001192092896 & Purity/4FRt0SRqqt0FBt/FRt_purity0 :"Purity/*FRt*SRqqt3FBt/event_purity0.08908046036958694 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 :"Purity/*FRt*SRqqt*FBt/event_purity0.20481795072555542 8 Purity/*FRt*SRqqt0FBt/FRt_purity0.16521237790584564 & Purity/3FRt0SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.4305555522441864 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_5_of_60 & Purity/1FRt*SRqqt4FBt/FRt_purity0 7 Purity/*FRt1SRqqt*FBt/FBt_purity0.9841686487197876 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 ("Purity/1FRt2SRqqt0FBt/event_purity0 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 +"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.25 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9690475463867188 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 ;"Purity/2FRt0SRqqt*FBt/event_purity0.014877114444971085 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 particle/accuracy_4_of_71 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 *jet/accuracy_2_of_40.2762931287288666 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.3700048625469208 particle/accuracy_2_of_21 ("Purity/*FRt4SRqqt*FBt/event_purity0 & Purity/*FRt4SRqqt*FBt/FBt_purity1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.3565407991409302 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9574074149131775 :"Purity/2FRt0SRqqt1FBt/event_purity0.01944444514811039 7 Purity/2FRt0SRqqt1FBt/FBt_purity0.9916666746139526 )jet/accuracy_1_of_50.918146550655365 & Purity/*FRt0SRqqt4FBt/FRt_purity0 ("Purity/0FRt0SRqqt0FBt/event_purity1 5 Purity/1FRt0SRqqt*FBt/FRt_purity0.22305928170681 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.36798471212387085 particle/accuracy_1_of_21 ;"Purity/*FRt1SRqqt2FBt/event_purity0.020737329497933388 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 jet/accuracy_5_of_50 particle/accuracy_1_of_11 *"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.37597399950027466 *jet/accuracy_1_of_10.4200733006000519 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9583333134651184 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 9"Purity/0FRt0SRqqt3FBt/event_purity0.1733333170413971 ("Purity/2FRt1SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt0FBt/event_purity0 ("Purity/0FRt4SRqqt*FBt/event_purity0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.2857142984867096 ("Purity/4FRt0SRqqt*FBt/event_purity0 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.1388888955116272 ("Purity/2FRt3SRqqt1FBt/event_purity0 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.18958131968975067 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.3994259238243103 ;"Purity/1FRt*SRqqt1FBt/event_purity0.058573395013809204 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.38349446654319763 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt3SRqqt*FBt/event_purity0 8 Purity/2FRt3SRqqt*FBt/FRt_purity0.18909522891044617 + Purity/0FRt*SRqqt4FBt/FBt_purity0.8125 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 9"Purity/*FRt0SRqqt3FBt/event_purity0.1011904701590538 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 :"Purity/1FRt*SRqqt*FBt/event_purity0.06555938720703125 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 ;"Purity/2FRt0SRqqt0FBt/event_purity0.010406260378658772 particle/accuracy_2_of_51 jet/accuracy_6_of_60 6 Purity/*FRt*SRqqt*FBt/FBt_purity0.984567403793335 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.36806002259254456 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.17888912558555603 & Purity/3FRt*SRqqt1FBt/FBt_purity1 particle/accuracy_4_of_41 & Purity/0FRt3SRqqt1FBt/FBt_purity1 8 Purity/*FRt3SRqqt1FBt/FRt_purity0.22777779400348663 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 9"Purity/1FRt0SRqqt3FBt/event_purity0.0357142873108387 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 particle/accuracy_2_of_41 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9746752381324768 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.25264546275138855 7 Purity/2FRt0SRqqt2FBt/FRt_purity0.3738095462322235 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.3350158929824829 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 jet/accuracy_7_of_70 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9979397654533386 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.3820827603340149 + Purity/1FRt*SRqqt3FBt/FRt_purity0.4375 :"Purity/1FRt0SRqqt*FBt/event_purity0.13588568568229675 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.27037036418914795 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 8 Purity/2FRt3SRqqt1FBt/FRt_purity0.15000000596046448 9"Purity/*FRt0SRqqt*FBt/event_purity0.3486631214618683 7 Purity/*FRt3SRqqt*FBt/FRt_purity0.1674565225839615 6 Purity/0FRt2SRqqt*FBt/FBt_purity0.984639048576355 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 7 Purity/0FRt2SRqqt2FBt/FBt_purity0.9615384340286255 ("Purity/1FRt2SRqqt2FBt/event_purity0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 ("Purity/0FRt3SRqqt1FBt/event_purity0 7 Purity/1FRt1SRqqt0FBt/FRt_purity0.2018723040819168 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 8 Purity/2FRt2SRqqt1FBt/FRt_purity0.25654762983322144 particle/accuracy_1_of_41 7 Purity/*FRt0SRqqt4FBt/FBt_purity0.8500000238418579 epoch0 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 *jet/accuracy_2_of_60.4561403691768646 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity0.875 & Purity/4FRt*SRqqt0FBt/FRt_purity0 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 :"Purity/0FRt1SRqqt*FBt/event_purity0.29666027426719666 &particle/f_score0.286234974861145 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.46666666865348816 9"Purity/0FRt0SRqqt*FBt/event_purity0.8151975274085999 ,jet/accuracy_4_of_40.004463689401745796 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_6_of_61 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 :"Purity/0FRt1SRqqt1FBt/event_purity0.09893309324979782 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9918307662010193 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 & Purity/2FRt*SRqqt3FBt/FRt_purity1 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.46828174591064453 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9967661499977112 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.14467592537403107 ,particle/sensitivity0.21157196164131165 ("Purity/1FRt0SRqqt4FBt/event_purity0 :"Purity/0FRt2SRqqt1FBt/event_purity0.01075268816202879 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.33852043747901917 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 ("Purity/3FRt0SRqqt1FBt/event_purity0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.8835318684577942 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity0.3206464946269989 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ("Purity/*FRt0SRqqt4FBt/event_purity0 8 Purity/*FRt0SRqqt1FBt/FRt_purity0.34754031896591187 ("Purity/1FRt3SRqqt0FBt/event_purity0 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 jet/accuracy_2_of_70 & Purity/2FRt*SRqqt3FBt/FBt_purity1 & Purity/3FRt0SRqqt2FBt/FBt_purity1 ("Purity/4FRt*SRqqt*FBt/event_purity0 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2465856820344925 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.32545289397239685 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 8 Purity/2FRt1SRqqt*FBt/FRt_purity0.21749290823936462 particle/accuracy_3_of_51 particle/accuracy_6_of_71 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.39191603660583496 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9910756349563599 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.8750000596046448 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.3300520181655884 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 7 Purity/2FRt0SRqqt2FBt/FBt_purity0.9940475821495056 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.450892835855484 ("Purity/3FRt1SRqqt*FBt/event_purity0 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 ("Purity/*FRt3SRqqt0FBt/event_purity0 9"Purity/0FRt*SRqqt1FBt/event_purity0.6166820526123047 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.3975643217563629 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity0 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.36260172724723816 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.25555554032325745 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 +jet/accuracy_2_of_20.07067140191793442 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 :"Purity/*FRt1SRqqt0FBt/event_purity0.19537661969661713 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.5 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 *jet/accuracy_1_of_40.8650160431861877 *jet/accuracy_2_of_50.3450578451156616 & Purity/*FRt4SRqqt1FBt/FBt_purity1 & Purity/4FRt2SRqqt*FBt/FRt_purity0 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.17273478209972382 8 Purity/1FRt*SRqqt*FBt/FRt_purity0.21822147071361542 ("Purity/1FRt3SRqqt*FBt/event_purity0 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.34431639313697815 ("Purity/3FRt0SRqqt0FBt/event_purity0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.38271602988243103 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ;"Purity/1FRt1SRqqt*FBt/event_purity0.010729643516242504 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.16644905507564545 & Purity/4FRt1SRqqt*FBt/FRt_purity0 ("Purity/4FRt1SRqqt0FBt/event_purity0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.3075000047683716 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9927419424057007 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.5 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 ("Purity/1FRt2SRqqt1FBt/event_purity0 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 ("Purity/2FRt4SRqqt0FBt/event_purity0 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.3055555522441864 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8868313431739807 8 Purity/3FRt*SRqqt0FBt/FRt_purity0.13566139340400696 & Purity/3FRt0SRqqt*FBt/FBt_purity1 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.1666666716337204 +jet/accuracy_3_of_60.07017543911933899 trainer/global_step603 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.21987119317054749 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 6 Purity/*FRt1SRqqt1FBt/FBt_purity0.994231641292572 ;"Purity/1FRt1SRqqt0FBt/event_purity0.012278445065021515 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity0.20726962387561798 ("Purity/0FRt3SRqqt0FBt/event_purity0 & Purity/1FRt0SRqqt4FBt/FRt_purity0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity0 9"Purity/*FRt*SRqqt0FBt/event_purity0.1390431523323059 :"Purity/*FRt1SRqqt*FBt/event_purity0.15690869092941284 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.0476190485060215 )loss/val_total_loss49.97751034713482 ;"Purity/*FRt1SRqqt1FBt/event_purity0.054934509098529816 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 particle/accuracy_5_of_51 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9787802696228027 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.375 _runtime1065.341910754 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9951668977737427 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity1 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.12745098769664764 particle/accuracy_2_of_71 & Purity/0FRt3SRqqt*FBt/FBt_purity1 % _timestamp1.7884661752978544e+09 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.3205128014087677 ("Purity/1FRt*SRqqt4FBt/event_purity0 9"Purity/0FRt*SRqqt2FBt/event_purity0.2613388001918793 8 Purity/2FRt0SRqqt0FBt/FRt_purity0.14929743111133575 :"Purity/2FRt0SRqqt2FBt/event_purity0.02500000037252903 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ("Purity/2FRt1SRqqt1FBt/event_purity0 ("Purity/2FRt3SRqqt0FBt/event_purity0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.3368164896965027 7 Purity/1FRt1SRqqt2FBt/FRt_purity0.4009353816509247 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.6190475821495056 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.4125337302684784 particle/accuracy_1_of_51 ."Purity/*FRt4SRqqt0FBt/SRqqt_purity0.34375 particle/accuracy_3_of_71 ("Purity/4FRt1SRqqt*FBt/event_purity0 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.41132956743240356 8 Purity/*FRt1SRqqt2FBt/FRt_purity0.35067036747932434 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 8 Purity/3FRt1SRqqt*FBt/FRt_purity0.18930043280124664 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ("Purity/*FRt3SRqqt*FBt/event_purity0 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9967741966247559 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/1FRt0SRqqt4FBt/FBt_purity1 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.3859025239944458 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9842396974563599 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 & Purity/3FRt*SRqqt2FBt/FBt_purity1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ("Purity/1FRt4SRqqt*FBt/event_purity0 _step48 9"Purity/0FRt*SRqqt3FBt/event_purity0.1358024626970291 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9953917264938354 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.4285714328289032 <"Purity/1FRt1SRqqt2FBt/event_purity0.0071428571827709675 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 8 Purity/2FRt*SRqqt2FBt/FRt_purity0.39000001549720764 & Purity/3FRt1SRqqt1FBt/FBt_purity1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.3382842242717743 particle/accuracy_7_of_70.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity0 ("Purity/*FRt1SRqqt3FBt/event_purity0 8 Purity/*FRt0SRqqt2FBt/FRt_purity0.40742093324661255 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 (particle/accuracy0.8352339267730713 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 :"Purity/1FRt0SRqqt0FBt/event_purity0.14258244633674622 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.588888943195343 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.17743654549121857 particle/accuracy_3_of_61 jet/accuracy_6_of_70 & Purity/1FRt4SRqqt0FBt/FRt_purity0 ("Purity/0FRt1SRqqt3FBt/event_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 ;"Purity/2FRt*SRqqt2FBt/event_purity0.017777778208255768 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 +jet/accuracy_2_of_30.18297651410102844 8 Purity/2FRt4SRqqt*FBt/FRt_purity0.30000001192092896 & Purity/2FRt2SRqqt2FBt/FBt_purity1 & Purity/4FRt0SRqqt*FBt/FRt_purity0 :"Purity/*FRt0SRqqt2FBt/event_purity0.26113954186439514 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 jet/accuracy_4_of_50 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity0 & Purity/1FRt3SRqqt2FBt/FRt_purity1 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8837166428565979 & Purity/*FRt3SRqqt*FBt/FBt_purity1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.35856306552886963 7 Purity/1FRt*SRqqt2FBt/FRt_purity0.4352588653564453 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.32949307560920715 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 7 Purity/*FRt2SRqqt2FBt/FRt_purity0.3205128312110901 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32400012016296387 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 particle/accuracy_5_of_61 jet/accuracy_4_of_70 & Purity/2FRt0SRqqt3FBt/FRt_purity1 :"Purity/*FRt0SRqqt0FBt/event_purity0.12108093500137329 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.37786826491355896 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.4000000059604645 ("Purity/0FRt4SRqqt0FBt/event_purity0 & Purity/2FRt3SRqqt*FBt/FBt_purity1 & Purity/*FRt3SRqqt2FBt/FBt_purity1 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 9"Purity/0FRt0SRqqt1FBt/event_purity0.9979397654533386 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt*SRqqt3FBt/event_purity0 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.21315227448940277 :"Purity/0FRt2SRqqt*FBt/event_purity0.04218094423413277 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 7 Purity/2FRt2SRqqt*FBt/FBt_purity0.9821428656578064 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.44736841320991516 & Purity/0FRt4SRqqt*FBt/FBt_purity1 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 :"Purity/*FRt2SRqqt0FBt/event_purity0.01924903318285942 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 ("Purity/0FRt3SRqqt*FBt/event_purity0 particle/accuracy_1_of_61 ("Purity/*FRt1SRqqt4FBt/event_purity0 9"Purity/0FRt0SRqqt2FBt/event_purity0.3327980935573578 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.14258244633674622 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 :"Purity/*FRt*SRqqt2FBt/event_purity0.19419775903224945 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.24562212824821472 ;"Purity/2FRt*SRqqt0FBt/event_purity0.003246735082939267 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.34561991691589355 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 ;"Purity/*FRt2SRqqt1FBt/event_purity0.004378923214972019 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9925950169563293 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4279800355434418 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9947412610054016 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.3674633204936981 ("Purity/3FRt0SRqqt*FBt/event_purity0 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.1388888955116272 7 Purity/0FRt1SRqqt3FBt/FBt_purity0.8666666150093079 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.40637654066085815 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 +particle/specificity0.9506799578666687 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.3499999940395355 particle/accuracy_1_of_71 ("Purity/*FRt2SRqqt3FBt/event_purity0 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9836632013320923 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ("Purity/4FRt2SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt1FBt/FRt_purity0.2983627915382385 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.3935238718986511 6 Purity/2FRt2SRqqt0FBt/FRt_purity0.151878222823143 ("Purity/0FRt*SRqqt4FBt/event_purity0 7"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.69047611951828 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity0 ("Purity/3FRt*SRqqt0FBt/event_purity0 + Purity/0FRt0SRqqt4FBt/FBt_purity0.8125 ("Purity/3FRt4SRqqt*FBt/event_purity0 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.9166666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.22454558312892914 ."Purity/1FRt*SRqqt3FBt/event_purity0.03125 7 Purity/2FRt*SRqqt1FBt/FBt_purity0.9910394549369812 8 Purity/2FRt3SRqqt0FBt/FRt_purity0.21369047462940216 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9962766766548157 :"Purity/0FRt*SRqqt*FBt/event_purity0.40748661756515503 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 :"Purity/1FRt*SRqqt0FBt/event_purity0.06467058509588242 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 +jet/accuracy_3_of_40.02782217226922512 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.20000000298023224 7 Purity/*FRt3SRqqt0FBt/FRt_purity0.1729462444782257 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9952755570411682 :"Purity/1FRt*SRqqt2FBt/event_purity0.10495587438344955 ("Purity/2FRt1SRqqt*FBt/event_purity0 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 ("Purity/3FRt2SRqqt0FBt/event_purity0 jet/accuracy_1_of_71 ("Purity/1FRt4SRqqt0FBt/event_purity0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.3713327944278717 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.35863494873046875 7 Purity/1FRt1SRqqt*FBt/FRt_purity0.2292298525571823 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 & Purity/3FRt2SRqqt1FBt/FBt_purity1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity0.25 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 :"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.43421053886413574 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 particle/accuracy_2_of_31 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8968254327774048 ("Purity/3FRt3SRqqt*FBt/event_purity0 8 Purity/*FRt1SRqqt1FBt/FRt_purity0.31259778141975403 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.3663094937801361 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.4014635682106018 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.6190475821495056 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 ("Purity/*FRt3SRqqt1FBt/event_purity0 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.39810892939567566 +jet/accuracy_3_of_50.06857157498598099 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9798036217689514 ("Purity/1FRt3SRqqt1FBt/event_purity0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 particle/accuracy_4_of_51 ("Purity/1FRt1SRqqt4FBt/event_purity0 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 ("Purity/3FRt*SRqqt*FBt/event_purity0 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 ("Purity/2FRt4SRqqt*FBt/event_purity0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2916666567325592 & Purity/4FRt*SRqqt*FBt/FRt_purity0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 9"Purity/1FRt0SRqqt1FBt/event_purity0.1138874739408493 8 Purity/1FRt0SRqqt2FBt/FRt_purity0.44340893626213074 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.3943653702735901 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 *jet/accuracy_1_of_30.8152989149093628 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 ("Purity/2FRt2SRqqt1FBt/event_purity0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 particle/accuracy_1_of_31 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.38908833265304565 8 Purity/3FRt*SRqqt1FBt/FRt_purity0.30909091234207153 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.13405796885490417 ("Purity/0FRt2SRqqt2FBt/event_purity0 9"Purity/*FRt0SRqqt1FBt/event_purity0.6299179196357727 & Purity/*FRt3SRqqt1FBt/FBt_purity1 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 ;"Purity/1FRt1SRqqt1FBt/event_purity0.005673849955201149 jet/accuracy_5_of_70 ("Purity/2FRt2SRqqt2FBt/event_purity0 ;"Purity/*FRt2SRqqt*FBt/event_purity0.016647348180413246 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.40642601251602173 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 & Purity/3FRt1SRqqt*FBt/FBt_purity1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.2587719261646271 ("Purity/*FRt3SRqqt2FBt/event_purity0 8 Purity/2FRt1SRqqt0FBt/FRt_purity0.17026859521865845 jet/accuracy_4_of_60 ( Purity/*FRt1SRqqt3FBt/FRt_purity0.5 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 :"Purity/1FRt0SRqqt2FBt/event_purity0.15711678564548492 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.3710061311721802 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.15687404572963715 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 & Purity/4FRt1SRqqt0FBt/FRt_purity0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.37181708216667175 ( Purity/1FRt1SRqqt3FBt/FRt_purity0.5 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.36909013986587524 ("Purity/2FRt1SRqqt2FBt/event_purity0 & Purity/*FRt*SRqqt4FBt/FRt_purity0 & Purity/4FRt2SRqqt0FBt/FRt_purity0 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.15067921578884125 7 Purity/1FRt*SRqqt0FBt/FRt_purity0.1686033457517624 ("Purity/3FRt4SRqqt0FBt/event_purity0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt0FBt/FRt_purity0.1462327092885971 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.8999999761581421 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ;"Purity/2FRt*SRqqt*FBt/event_purity0.005416110623627901 ("Purity/3FRt1SRqqt1FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.5 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 7 Purity/2FRt*SRqqt0FBt/FRt_purity0.1641518771648407 ("Purity/*FRt4SRqqt0FBt/event_purity0 & Purity/2FRt3SRqqt1FBt/FBt_purity1 ("Purity/2FRt0SRqqt3FBt/event_purity0 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 ("Purity/3FRt3SRqqt0FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 & Purity/0FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt0SRqqt2FBt/event_purity0 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity0.027037035673856735 ,&Purity/0FRt4SRqrqt2FBt/event_proportion0 7 Purity/2FRt2SRqqt1FBt/FBt_purity0.9821428656578064 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.3586941063404083 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.18539854884147644 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ("Purity/3FRt2SRqqt1FBt/event_purity0 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.2977248728275299 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.4177035391330719 8 Purity/3FRt0SRqqt*FBt/FRt_purity0.23232322931289673 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9845708012580872 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.37401270866394043 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ("Purity/1FRt2SRqqt*FBt/event_purity0 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 particle/accuracy_3_of_31 ("Purity/3FRt*SRqqt2FBt/event_purity0 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.04761904850602150 w (:  disk.sda3.in0' disk.nvme0c0n1.out22191.9130859375 disk.sda2.in0# disk.nvme0c0n1.in 2464.21484375 disk.dm-1.in0 disk.drbd1260.out 0.1484375( proc.memory.availableMB 3630.9921875# proc.memory.rssMB 3166.55078125 cpu24.421856432472026 disk.sda3.out 33.07421875 disk.sda.in0) disk./.usagePercent77.05761857122071% disk.nvme0n1.out22191.9130859375 proc.cpu.threads17 network.sent 1.004206e+06! disk.nvme0n1.in 2464.21484375 disk.sda1.in0 disk.sda2.out0 disk.dm-1.out 33.07421875 disk.drbd1260.in0$ memory_percent77.83818244934082* proc.memory.percent19.327092170715332 disk.sda.out 33.07421875 network.recv390446% disk./.usageGB1148.2142028808594 disk.dm-0.in 2464.21484375" disk.dm-0.out22191.9130859375 disk.sda1.out08 V (:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp38.0 gpu.0.powerWatts119.857' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.971400000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0M͓ (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp44.0 gpu.0.powerWatts227.354' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.4708 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0L t (: ٣* proc.memory.percent19.327497482299805 cpu24.435064656233447) disk./.usagePercent77.06144076370694# disk.nvme0c0n1.in 2464.21484375 disk.dm-1.in0 disk.drbd1260.out 0.1484375! disk.nvme0n1.in 2464.21484375% disk.nvme0n1.out23755.9873046875 disk.dm-0.in 2464.21484375 disk.sda1.in0 network.sent 1.109201e+06 disk.dm-1.out 33.5546875 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda2.in0$ memory_percent77.83606052398682" proc.memory.rssMB 3166.6171875% disk./.usageGB1148.2711563110352 disk.drbd1260.in0" disk.dm-0.out23755.9873046875 disk.sda3.out 33.5546875 disk.sda.out 33.5546875 disk.sda2.out0) proc.memory.availableMB 3631.33984375 proc.cpu.threads17 network.recv396495' disk.nvme0c0n1.out23755.9873046875E} (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts230.054' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.0108 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0\t (:  disk.drbd1260.out 0.15234375$ memory_percent78.39446067810059( proc.memory.availableMB 3539.8515625" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 cpu24.44734790982578 network.sent 1.114885e+06 network.recv400719! disk.nvme0n1.in 2590.27734375# disk.nvme0c0n1.in 2590.27734375 proc.cpu.threads17$ disk./.usageGB1148.459156036377# disk.dm-0.out23815.12060546875 disk.sda1.in0 disk.sda.in0 disk.sda.out 33.9765625 disk.sda2.out0 disk.dm-1.out 33.9765625 disk.drbd1260.in0& disk.nvme0n1.out23815.12060546875 disk.dm-0.in 2590.27734375 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 33.9765625 disk.sda2.in0( disk.nvme0c0n1.out23815.12060546875 disk.dm-1.in0( disk./.usagePercent77.0740575830169 RH (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts246.885' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.376999999999995 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0׏nu (:  network.sent 1.120011e+06 disk.drbd1260.in0# disk.dm-0.out24138.35107421875 disk.sda1.out0 disk.sda.out 34.3828125# disk.nvme0c0n1.in 2590.27734375* proc.memory.percent19.321489334106445 cpu24.459646554923477 network.recv404226! disk.nvme0n1.in 2590.27734375 disk.sda3.in0 disk.sda2.in0 proc.cpu.threads17$ disk./.usageGB1148.435474395752 disk.dm-1.out 34.3828125& disk.nvme0n1.out24138.35107421875 disk.sda3.out 34.3828125 disk.sda2.out0$ memory_percent78.95879745483398' proc.memory.availableMB 3447.390625) disk./.usagePercent77.07246828824434 disk.dm-1.in0 disk.drbd1260.out 0.15234375 disk.dm-0.in 2590.27734375 disk.sda1.in0 disk.sda.in0( disk.nvme0c0n1.out24138.35107421875" proc.memory.rssMB 3165.6328125ʞ (: ɨè gpu.0.gpu83.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp44.0 gpu.0.powerWatts239.642' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.928399999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0s^t (: ɨ» proc.cpu.threads17 network.recv408801% disk./.usageGB1148.6630058288574% disk.nvme0n1.out24412.8486328125 disk.dm-0.in 2590.27734375 disk.sda1.out0$ memory_percent79.59954738616943) proc.memory.availableMB 3342.41015625" proc.memory.rssMB 3165.6328125! disk.nvme0n1.in 2590.27734375 disk.sda1.in0 disk.sda3.in0 disk.sda.out 34.80078125 disk.sda2.in0 disk.sda2.out0 disk.dm-1.out 34.80078125 disk.drbd1260.out0.15625 network.sent 1.126042e+06" disk.dm-0.out24412.8486328125 disk.sda.in0* proc.memory.percent19.321489334106445 cpu24.47148715913052) disk./.usagePercent77.08773811363163 disk.sda3.out 34.80078125# disk.nvme0c0n1.in 2590.27734375' disk.nvme0c0n1.out24412.8486328125 disk.dm-1.in0 disk.drbd1260.in0i%D2 - _wandbruntime1150  _runtime1150^  _runtime1150.149276368 _step49 & lr-AdamW/pg10.0011775546048562045 & lr-AdamW/pg20.0011775546048562045 trainer/global_step649 % _timestamp1.7884662601064315e+092)  _step49 & lr-AdamW/pg10.0011775546048562045 & lr-AdamW/pg20.0011775546048562045 trainer/global_step649 % _timestamp1.7884662601064315e+09 _runtime1150.1492763681`O^2 - _wandbruntime1150  _runtime1150' O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed141665247151104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated2886629 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed4584788 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1353325056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed21571098 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated9359102 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed135905927007728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed143172342996480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated7471833 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed7471414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884662601079957e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1507628663193 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1505114294789 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1507095845376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed12212002 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed2886626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1936023748096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated137413581098889 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated12231684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1509612207616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current52648960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 _runtime1150.149569901 _step50 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 trainer/global_step649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated143174884786688 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated9359102 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current419 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated135905952435696 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated4585204 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed12212002 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated143174884786688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed9359096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1507095845376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated12231684 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed9359096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed30921299066880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 409820160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed143172342996480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1509612207616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated21590786 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed141665247151104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated141665272579072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed28985327967744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated141665272579072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated28985357065728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current81746944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed143172342996480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1509612207616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated21590786 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed141665247151104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated141665272579072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed28985327967744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated141665272579072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated28985357065728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current81746944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884662601079957e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1507628663193 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1505114294789 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1507095845376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed12212002 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed2886626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 1936023748096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated137413581098889 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated12231684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1509612207616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current52648960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 _runtime1150.149569901 _step50 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 trainer/global_step649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated143174884786688 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated9359102 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current4192Cܪ2 - _wandbruntime1151  _runtime1151O*B M UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed9372977 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated137618382063535 % _timestamp1.7884662616518002e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated21623044 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated12250061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated143388150391808 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed4591723 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated136108481145280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current81733632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated30968358382592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 trainer/global_step649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated7483104 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated141876262954496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current52635648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pooϜtl.freed136108455717312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 _runtime 1151.69314305 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed143385608588288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed7482681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated9372983 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1938887311872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed9372977 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2890958 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated141876262954496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed21603330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current423 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed12230353 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed137615842266955 _step51 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1511887437312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed141876237526528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed12230353 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated2890961 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed21603330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated9372983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated12250061 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1509371061760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated21623044 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated21623044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated136108481145280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3200253952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current81733632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated30968358382592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 trainer/global_step649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated7483104 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated141876262954496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current52635648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed136108455717312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 _runtime 1151.69314305 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed143385608588288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed7482681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated9372983 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 1938887311872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed9372977 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed2890958 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated141876262954496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed21603330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current423 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed12230353 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed137615842266955 _step51 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1511887437312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed141876237526528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed12230353 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated2890961 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed2160333034E2 - _wandbruntime1151  _runtime1151nwxR  .loss/FRt1/assignment_loss1.48383480489254 /loss/FRt4/assignment_loss3.321997090052533 /loss/FBt1/detection_loss0.8581063285408332 /loss/FBt4/detection_loss2.6527092456817627 _step52 0loss/FBt1/assignment_loss0.1956894668285077 trainer/global_step649 .loss/FRt1/detection_loss1.777879305407405 epoch1 /loss/FRt3/assignment_loss4.456427751802931 1loss/SRqqt1/assignment_loss1.106689839746601 2loss/SRqqt2/assignment_loss2.2262423042207957 2loss/SRqqt3/assignment_loss2.4055457735061645 1loss/FBt2/assignment_loss0.19804347023326793 0loss/FBt3/assignment_loss0.2596841880494091 .loss/FBt2/detection_loss3.042748353060554 %loss/total_loss48.18221831668485 0loss/FBt4/assignment_loss0.3056619588017423 /loss/FRt2/detection_loss2.3857904368190357 .loss/FRt4/detection_loss2.420334150393804 1loss/SRqqt1/detection_loss1.1338281711173612 1loss/SRqqt4/detection_loss1.9825236849719658 .loss/FRt3/detection_loss3.172554976799909 0loss/SRqqt2/detection_loss2.232157575711608 1loss/SRqqt3/detection_loss2.3306229197978974 _runtime1151.694545543 /loss/FRt2/assignment_loss3.634669445535188 1loss/SRqqt4/assignment_loss1.877196396390597 /loss/FBt3/detection_loss2.7212806783224406 % _timestamp1.7884662616539023e+09#V  .loss/FRt3/detection_loss3.172554976799909 0loss/SRqqt2/detection_loss2.232157575711608 1loss/SRqqt3/detection_loss2.3306229197978974 _runtime1151.694545543 /loss/FRt2/assignment_loss3.634669445535188 1loss/SRqqt4/assignment_loss1.877196396390597 /loss/FBt3/detection_loss2.7212806783224406 % _timestamp1.7884662616539023e+09 .loss/FRt1/assignment_loss1.48383480489254 /loss/FRt4/assignment_loss3.321997090052533 /loss/FBt1/detection_loss0.8581063285408332 /loss/FBt4/detection_loss2.6527092456817627 _step52 0loss/FBt1/assignment_loss0.1956894668285077 trainer/global_step649 .loss/FRt1/detection_loss1.777879305407405 epoch1 /loss/FRt3/assignment_loss4.456427751802931 1loss/SRqqt1/assignment_loss1.106689839746601 2loss/SRqqt2/assignment_loss2.2262423042207957 2loss/SRqqt3/assignment_loss2.4055457735061645 1loss/FBt2/assignment_loss0.19804347023326793 0loss/FBt3/assignment_loss0.2596841880494091 .loss/FBt2/detection_loss3.042748353060554 %loss/total_loss48.18221831668485 0loss/FBt4/assignment_loss0.3056619588017423 /loss/FRt2/detection_loss2.3857904368190357 .loss/FRt4/detection_loss2.420334150393804 1loss/SRqqt1/detection_loss1.1338281711173612 1loss/SRqqt4/detection_loss1.98252368497196584. (: ب gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp45.0 gpu.0.powerWatts224.977' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent44.995400000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0~Bn (: بڴ$ disk./.usageGB1148.662166595459' disk.nvme0c0n1.out24722.5048828125 disk.dm-1.out 35.2109375! disk.nvme0n1.in 2590.27734375 disk.sda1.in0" proc.memory.rssMB 3165.6328125 cpu24.4828939998419 network.sent 1.141767e+06) disk./.usagePercent77.08768179197409 disk.sda3.out 35.2109375 disk.drbd1260.in0 disk.drbd1260.out0.15625% disk.nvme0n1.out24722.5048828125* proc.memory.percent19.321489334106445 proc.cpu.threads17 network.recv411907 disk.sda.in0 disk.sda.out 35.2109375# disk.nvme0c0n1.in 2590.27734375 disk.sda1.out0$ memory_percent80.23552894592285( proc.memory.availableMB 3238.2109375 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.dm-0.in 2590.27734375" disk.dm-0.out24722.5048828125 _ (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts218.795' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent43.759 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0*f (: * proc.memory.percent19.321489334106445 cpu24.49384537984078& disk.nvme0n1.out24937.99267578125 disk.dm-1.in0 disk.drbd1260.out 0.16015625 proc.cpu.threads17 network.sent 1.215731e+06 disk.nvme0n1.in2696.625 disk.dm-0.in2696.625# disk.dm-0.out24937.99267578125 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0( disk.nvme0c0n1.out24937.99267578125 network.recv417853) disk./.usagePercent77.08925291021173% disk./.usageGB1148.6855773925781 disk.sda.in0 disk.sda.out 35.7421875 disk.sda2.out0 disk.drbd1260.in0" proc.memory.rssMB 3165.6328125 disk.sda1.in0 disk.sda3.out 35.7421875 disk.nvme0c0n1.in2696.625 disk.dm-1.out 35.7421875$ memory_percent80.92222213745117' proc.memory.availableMB 3125.703125_{ (: Ө gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp43.0 gpu.0.powerWatts161.461' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent32.2922 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0'Qj (:  network.recv421406% disk./.usageGB1148.6139068603516 disk.dm-1.out 36.12890625 disk.dm-0.in2696.625 disk.sda1.in0) proc.memory.availableMB 3013.66796875" proc.memory.rssMB 3165.6328125 cpu24.5050032897721 network.sent 1.221393e+06( disk.nvme0c0n1.out25284.78759765625 disk.dm-1.in0 disk.sda3.in0 disk.sda3.out 36.12890625 disk.sda.in0 disk.sda2.in0$ memory_percent81.60603046417236) disk./.usagePercent77.08444304065843 disk.nvme0c0n1.in2696.625 disk.sda1.out0 disk.sda.out 36.12890625 disk.sda2.out0* proc.memory.percent19.321489334106445 proc.cpu.threads17 disk.drbd1260.in0 disk.drbd1260.out 0.16015625 disk.nvme0n1.in2696.625& disk.nvme0n1.out25284.78759765625# disk.dm-0.out25284.78759765625b (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts235.401' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.0802 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors01i (: ) disk./.usagePercent77.11655892982134 disk.sda3.out 36.53515625 disk.sda.in0 disk.sda.out 36.53515625 disk.sda2.in0 disk.sda2.out0& disk.nvme0n1.out25670.75830078125) proc.memory.availableMB 2897.12109375$ disk./.usageGB1149.092456817627( disk.nvme0c0n1.out25670.75830078125 disk.dm-0.in2696.625# disk.dm-0.out25670.75830078125 disk.sda1.in0 disk.sda1.out0" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 network.recv425689 disk.sda3.in0 disk.nvme0c0n1.in2696.625 disk.dm-1.out 36.53515625 disk.nvme0n1.in2696.625 network.sent 1.228726e+06 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.1640625$ memory_percent82.31737613677979 cpu24.51558701075586 proc.cpu.threads17K (:  gpu.0.gpu81.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp49.0 gpu.0.powerWatts338.483' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent67.69659999999999 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0{f (:  disk.dm-1.out 36.9296875 disk.sda2.in0 disk.sda2.out0$ memory_percent82.95350074768066( proc.memory.availableMB 2792.8984375* proc.memory.percent19.321489334106445 network.recv429523( disk.nvme0c0n1.out26017.07568359375 disk.dm-0.in2696.625 disk.sda1.in0 disk.sda3.in0" proc.memory.rssMB 3165.6328125 proc.cpu.threads17 disk.drbd1260.out 0.1640625 disk.nvme0n1.in2696.625& disk.nvme0n1.out26017.07568359375 disk.sda1.out0 cpu24.526250532687342 network.sent 1.23426e+06) disk./.usagePercent77.11826317197676 disk.dm-1.in0 disk.drbd1260.in0# disk.dm-0.out26017.07568359375 disk.sda3.out 36.9296875 disk.sda.in0 disk.sda.out 36.9296875% disk./.usageGB1149.1178512573242 disk.nvme0c0n1.in2696.625i# (:  gpu.0.gpu43.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp49.0 gpu.0.powerWatts333.193' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.6386 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Zn (:  disk.sda.out 37.36328125% disk.nvme0n1.out26465.0634765625 disk.dm-0.in 2698.2734375" proc.memory.rssMB 3165.6328125" disk.nvme0c0n1.in 2698.2734375 disk.dm-1.in0" disk.dm-0.out26465.0634765625 proc.cpu.threads17' disk.nvme0c0n1.out26465.0634765625 disk.sda1.in0 disk.sda3.out 37.36328125 disk.sda2.out0 disk.dm-1.out 37.36328125 disk.drbd1260.in0 disk.nvme0n1.in 2698.2734375# memory_percent83.5622787475586( disk./.usagePercent77.1182521636528% disk./.usageGB1149.1176872253418 disk.sda3.in0 disk.sda2.in0 disk.drbd1260.out0.171875& proc.memory.availableMB 2693.15625* proc.memory.percent19.321489334106445 cpu24.536387361031554 network.sent 1.240037e+06 network.recv433853 disk.sda1.out0 disk.sda.in092 - _wandbruntime1232  _runtime1232,:  _runtime1232.902272621 _step53 & lr-AdamW/pg10.0011768664283972941 & lr-AdamW/pg20.0011768664283972941 trainer/global_step699 % _timestamp1.7884663428596165e+09W]  & lr-AdamW/pg10.0011768664283972941 & lr-AdamW/pg20.0011768664283972941 trainer/global_step699 % _timestamp1.7884663428596165e+09 _runtime1232.902272621 _step535nS2 - _wandbruntime1232  _runtime1232( O [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1623400855040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated146032387914896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated152214791350272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed8033734 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed3103226 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated33270274732032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated31188425532928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed153835650415104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1620884492800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2081849199104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 _runtime1232.902423175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1367342592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 _step54 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated153838192205312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed153835650415104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated10053152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed152214765922304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed10053146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated152214791350272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated153838192205312 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2081790258688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated13152586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed4930508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.pKeak864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1618754209497 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed153835650415104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 1620884492800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2081849199104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 _runtime1232.902423175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1367342592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 _step54 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated153838192205312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed153835650415104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated10053152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed152214765922304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed10053146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated152214791350272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated153838192205312 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2081790258688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated13152586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1526 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed4930508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1618754209497 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 $ _timestamp1.788466344393482e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated152425781725696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated13170963 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated4937818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2084774316544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated10067033 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1623159709184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed146234891196512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated13170963 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated23237996 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated8045379 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed8044947 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 trainer/global_step699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1625676084736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated154051457810432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated152425781725696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed152425756297728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed23218282 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated23237996 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed31232457804288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3107558 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated147858457457443 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated154051457810432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated146234916624480 M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed10067027 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed13151255 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1367342592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 $ _timestamp1.788466344393482e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated152425781725696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated13170963 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated4937818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2084774316544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated10067033 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1623159709184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed146234891196512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated13170963 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated23237996 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated8045379 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed8044947 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 trainer/global_step699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1625676084736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated154051457810432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated152425781725696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed152425756297728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed23218282 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated23237996 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed31232457804288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3107558 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated147858457457443 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated154051457810432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated146234916624480 P2 - _wandbruntime1315  _runtime1315H  & lr-AdamW/pg10.0011756873211670165 & lr-AdamW/pg20.0011756873211670165 trainer/global_step749 $ _timestamp1.788466425429215e+09 _runtime1315.469676124 _step57  _step57 & lr-AdamW/pg10.0011756873211670165 & lr-AdamW/pg20.0011756873211670165 trainer/global_step749 $ _timestamp1.788466425429215e+09 _runtime1315.4696761249Jo2 - _wandbruntime1315  _runtime1315E' O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated24820690 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated164501499623936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated8595837 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed10747196 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed10747196 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed14053806 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed162764284693504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated164501499623936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated5276008 _step58 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1734673140224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated33391494000128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated14073488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1737189502464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated157893731886705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed162764284693504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 trainer/global_step749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2227561431040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 417658368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed3319826 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed164498957833728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1737189502464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated162764310121472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed157891192090333 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2227502490624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed156158797966128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed164498957833728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated14073488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed35618967392768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated35619055431168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed8595375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1732394124205 $ _timestamp1.788466425430028e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3319829 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672-h' O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated35619055431168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed8595375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1732394124205 $ _timestamp1.788466425430028e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3319829 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated24820690 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated164501499623936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated8595837 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed10747196 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed10747196 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed14053806 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed162764284693504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated164501499623936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated5276008 _step58 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1734673140224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated33391494000128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated14073488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1737189502464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated157893731886705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed162764284693504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 trainer/global_step749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2227561431040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 417658368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed3319826 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed164498957833728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1737189502464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3200253952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated162764310121472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed157891192090333 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2227502490624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed156158797966128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed164498957833728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated14073488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed35618967392768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526: _2 - _wandbruntime1316  _runtime1316#fZ5 M J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed8606545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated8606996 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 TDDeviceSĹt$tatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1739464732160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1737180747671 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed10761077 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1736948356608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3324161 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated156361352103680 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated24852948 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated164714765229056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated33435555369472 % _timestamp1.7884664269033203e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed162975275068928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed10761077 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed33435526271488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed158095993054771 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1734666379059 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated162975300496896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 417658368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated14091865 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed164712223425536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated10761083 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88025088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2230470295552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5282387 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _step59 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current451 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 trainer/global_step749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3324158 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated10761083 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 _runtime1316.944338459J{& M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated162975300496896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 417658368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated14091865 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed164712223425536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated10761083 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88025088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2230470295552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5282387 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _step59 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current451 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 trainer/global_step749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3324158 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated10761083 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 _runtime1316.944338459 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed8606545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated8606996 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1739464732160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1737180747671 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed10761077 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1736948356608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3324161 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated156361352103680 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated24852948 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated164714765229056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated33435555369472 % _timestamp1.7884664269033203e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed162975275068928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed10761077 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed33435526271488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed158095993054771 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1734666379059 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672;$62 - _wandbruntime1316  _runtime1316ƀ[  1loss/FBt2/assignment_loss0.06892856988609614 1loss/SRqqt3/detection_loss2.5992371320724486 _runtime1316.946452605 /loss/FRt4/assignment_loss3.106190520681833 1loss/FBt3/assignment_loss0.18344246252672747 1loss/SRqqt1/detection_loss1.0662018429138698 1loss/SRqqt2/detection_loss2.0410626601766455 /loss/FBt1/detection_loss0.6832385556358632 trainer/global_step749 0loss/FRt1/assignment_loss1.3636002417692203 /loss/FBt2/detection_loss3.1928914388020835 1loss/FBt4/assignment_loss0.13486047903734288 /loss/FRt4/detection_loss2.6071400391428092 2loss/SRqqt1/assignment_loss1.0424259634455666 1loss/SRqqt2/assignment_loss2.390080027282238 /loss/FBt4/detection_loss2.5861069709062576 1loss/SRqqt3/assignment_loss2.019601251528813 2loss/SRqqt4/assignment_loss1.9102314421406357 %loss/total_loss46.69391318355589 .loss/FRt2/detection_loss2.577150153017592 .loss/FRt3/detection_loss2.913922456758363 _step60 0loss/FRt2/assignment_loss3.8022072253556085 0loss/FRt3/assignment_loss3.8796631193586757 1loss/FBt1/assignment_loss0.07966167446785055 /loss/FRt1/detection_loss1.6758184786309902 1loss/SRqqt4/detection_loss1.8941159334073785 .loss/FBt3/detection_loss2.876134544610977 epoch1 % _timestamp1.7884664269061542e+09HZ_  .loss/FBt3/detection_loss2.876134544610977 epoch1 % _timestamp1.7884664269061542e+09 1loss/FBt2/assignment_loss0.06892856988609614 1loss/SRqqt3/detection_loss2.5992371320724486 _runtime1316.946452605 /loss/FRt4/assignment_loss3.106190520681833 1loss/FBt3/assignment_loss0.18344246252672747 1loss/SRqqt1/detection_loss1.0662018429138698 1loss/SRqqt2/detection_loss2.0410626601766455 /loss/FBt1/detection_loss0.6832385556358632 trainer/global_step749 0loss/FRt1/assignment_loss1.3636002417692203 /loss/FBt2/detection_loss3.1928914388020835 1loss/FBt4/assignment_loss0.13486047903734288 /loss/FRt4/detection_loss2.6071400391428092 2loss/SRqqt1/assignment_loss1.0424259634455666 1loss/SRqqt2/assignment_loss2.390080027282238 /loss/FBt4/detection_loss2.5861069709062576 1loss/SRqqt3/assignment_loss2.019601251528813 2loss/SRqqt4/assignment_loss1.9102314421406357 %loss/total_loss46.69391318355589 .loss/FRt2/detection_loss2.577150153017592 .loss/FRt3/detection_loss2.913922456758363 _step60 0loss/FRt2/assignment_loss3.8022072253556085 0loss/FRt3/assignment_loss3.8796631193586757 1loss/FBt1/assignment_loss0.07966167446785055 /loss/FRt1/detection_loss1.6758184786309902 1loss/SRqqt4/detection_loss1.8941159334073785<F (: ΅ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts252.511' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent50.502199999999995 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0dkpz (: # disk.nvme0c0n1.in 2932.30078125 disk.drbd1260.in0$ memory_percent87.60199546813965 network.recv460461 disk.sda3.in0 disk.sda.in0 disk.sda.out 39.87890625 disk.sda2.in0 disk.dm-1.in0 disk.dm-1.out 39.87890625 disk.drbd1260.out 0.18359375) disk./.usagePercent77.08953298245419% disk./.usageGB1148.6897506713867 disk.sda1.in0 disk.sda1.out0 disk.sda3.out 39.87890625( disk.nvme0c0n1.out28245.27978515625( proc.memory.availableMB 2031.2890625 cpu24.591272611981804! disk.nvme0n1.in 2932.30078125& disk.nvme0n1.out28245.27978515625 disk.dm-0.in 2932.30078125 disk.sda2.out0" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 proc.cpu.threads17 network.sent 1.352984e+06# disk.dm-0.out28245.27978515625 w  (:  gpu.0.gpu61.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp50.0 gpu.0.powerWatts334.533' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent66.90660000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0A u (:  disk.sda3.in0 cpu24.599787369932027 network.recv466039) disk./.usagePercent77.01998162357438 disk.sda2.out0# disk.nvme0c0n1.in 2932.30078125 disk.dm-1.in0 disk.sda1.out0" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445% disk./.usageGB1147.6533851623535 disk.sda.in0 disk.sda2.in0 disk.drbd1260.in0& disk.nvme0n1.out28659.86181640625 disk.sda3.out 40.203125 proc.cpu.threads17 disk.sda.out 40.203125( disk.nvme0c0n1.out28659.86181640625 disk.dm-1.out 40.203125 disk.drbd1260.out 0.18359375 disk.dm-0.in 2932.30078125$ memory_percent88.23888301849365) proc.memory.availableMB 1926.94140625 network.sent 1.441761e+06! disk.nvme0n1.in 2932.30078125# disk.dm-0.out28659.86181640625 disk.sda1.in0o] (:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts240.205' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.041 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0q~=t (: ڎ disk.drbd1260.in0! disk.nvme0n1.in 2932.30078125 disk.sda2.out0* proc.memory.percent19.321489334106445 network.sent 1.447519e+06% disk./.usageGB1147.6533660888672 disk.dm-1.out 40.8984375& disk.nvme0n1.out28934.42529296875# disk.dm-0.out28934.42529296875 disk.sda1.out0 cpu24.607913308828156 network.recv470376# disk.nvme0c0n1.in 2932.30078125 disk.drbd1260.out0.1875 disk.dm-0.in 2932.30078125 disk.sda3.in0 disk.sda3.out 40.8984375 disk.sda.in0 disk.sda.out 40.8984375) disk./.usagePercent77.01998034353672( disk.nvme0c0n1.out28934.42529296875 disk.sda1.in0 disk.sda2.in0$ memory_percent88.84928226470947) proc.memory.availableMB 1826.93359375" proc.memory.rssMB 3165.6328125 proc.cpu.threads17 disk.dm-1.in0Ν (:  gpu.0.gpu73.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp47.0 gpu.0.powerWatts251.681' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent50.3362 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0m (:  network.recv473840( disk.nvme0c0n1.out29339.17529296875 disk.dm-0.in 3085.08203125 disk.sda1.in0 disk.sda1.out0 disk.sda.in0$ memory_percent89.44418430328369# disk.nvme0c0n1.in 3085.08203125 disk.sda2.out0" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 proc.cpu.threads17# disk.dm-0.out29339.17529296875 disk.sda3.in0 disk.sda3.out41.59375 disk.sda.out41.59375 disk.sda2.in0) proc.memory.availableMB 1729.46484375 cpu24.61607047133367) disk./.usagePercent77.00750509639317% disk./.usageGB1147.4674758911133 disk.dm-1.in0 disk.dm-1.out41.59375 disk.drbd1260.in0 disk.drbd1260.out0.1875! disk.nvme0n1.in 3085.08203125& disk.nvme0n1.out29339.17529296875 network.sent 1.450674e+06o[ (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts228.414' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent45.68279999999999 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0r (:  network.sent 1.45862e+06) disk./.usagePercent76.97716282742749% disk./.usageGB1147.0153541564941& disk.nvme0n1.out29701.74462890625 disk.sda1.out0( disk.nvme0c0n1.out29701.74462890625 disk.dm-1.in0 disk.dm-1.out 42.015625 network.recv478626! disk.nvme0n1.in 3085.08203125 disk.dm-0.in 3085.08203125 disk.sda3.out 42.015625 disk.sda.in0 disk.sda2.out0$ memory_percent90.03281593322754( proc.memory.availableMB 1633.0234375 disk.drbd1260.in0# disk.dm-0.out29701.74462890625 disk.sda1.in0 disk.sda.out 42.015625 disk.sda2.in0 proc.cpu.threads17 disk.drbd1260.out 0.1953125 disk.sda3.in0# disk.nvme0c0n1.in 3085.08203125" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 cpu24.623547878448953P2 - _wandbruntime1394  _runtime1394aCy  trainer/global_step799 % _timestamp1.7884665044896734e+09 _runtime1394.530588457 _step61 & lr-AdamW/pg10.0011740182676386686 & lr-AdamW/pg20.0011740182676386686 ב  _runtime1394.530588457 _step61 & lr-AdamW/pg10.0011740182676386686 & lr-AdamW/pg20.0011740182676386686 trainer/global_step799 % _timestamp1.7884665044896734e+09=&lb2 - _wandbruntime1394  _runtime1394ss  O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3200253952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed3536426 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1369439744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed35594533369344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated11441252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated175164807042560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 1848548407317 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated173313828892672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 % _timestamp1.7884665044905121e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed14974708 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3536429 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1850978149888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated168133807280613 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed175162265252352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated26435642 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated14994390 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed166285233445328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2373106467328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed26415954 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated37967668934656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 trainer/global_step799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current465 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 _step62 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed37967578799104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed175162265252352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed9157754 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed5621328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed173313803464704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed14974708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated14994390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1850978149888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated9158219 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated35594562467328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2373045429760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _runtime1394.531349562 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed173313803464704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated11441252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1848461787648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1846034038913 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated5621790 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated173313828892672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 % _timestamp1.7884665044905121e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed14974708 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3536429 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1850978149888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated168133807280613 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3200253952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed175162265252352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated26435642 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated14994390 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed166285233445328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2373106467328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3672 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed26415954 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated37967668934656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1526 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40261124096 trainer/global_step799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current465 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 _step62 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed37967578799104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed175162265252352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed9157754 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed5621328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1526 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed173313803464704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed14974708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40261124096 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated14994390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1850978149888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated9158219 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated35594562467328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2373045429760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _runtime1394.531349562 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed173313803464704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated11441252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 1848461787648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1846034038913 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated5621790 I0P (: Ȫʭ gpu.0.gpu97.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51472473144531* gpu.0.memoryAllocatedBytes 40814837760 gpu.0.temp46.0 gpu.0.powerWatts167.181' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent33.43620000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0{DQv (: Ȫۜ disk.sda1.in0 disk.sda2.in0( disk.nvme0c0n1.out29989.29931640625 disk.dm-1.out 42.62109375 disk.drbd1260.out 0.1953125" proc.memory.rssMB 3165.6328125* proc.memory.percent19.321489334106445 cpu24.631524207681537% disk./.usageGB1147.0156555175781 disk.sda3.out 42.62109375 disk.sda.in0 disk.sda2.out0# disk.nvme0c0n1.in 3085.08203125 disk.dm-1.in0$ memory_percent90.74573516845703& proc.memory.availableMB 1516.21875 network.sent 1.475306e+06 network.recv482334( disk./.usagePercent76.9771830520227 disk.dm-0.in 3085.08203125# disk.dm-0.out29989.29931640625 disk.sda1.out0 disk.sda.out 42.62109375 proc.cpu.threads17! disk.nvme0n1.in 3085.08203125& disk.nvme0n1.out29989.29931640625 disk.sda3.in0 disk.drbd1260.in0}2 - _wandbruntime1396  _runtime1396)J&& M _runtime1396.033838553 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3200253952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated166487787582880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40261124096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1526 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed11455127 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed11455127 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1369439744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated35638623836672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed35638594738688 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1850820662379 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated173524819268096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1848306293767 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 _step63 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed173524793840128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current463 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed168336068448679 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed26448186 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1850737004032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5628193 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2375954419712 % _timestamp1.7884665059927788e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3540758 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed9168951 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed173524793840128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated5628653 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed166487762154912 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1853253379584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated11455133 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated9169414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed14993059 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1850737004032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated175378072647680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1853253379584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated15012767 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated26467900 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3540761 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated38014637183488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated15012767 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated26467900 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3540761 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated38014637183488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3200253952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed11455127 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed11455127 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1369439744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated35638623836672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed35638594738688 KBDeviceStatsMonitor.on_train_batyfch_end/segment.small_pool.allocated1526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1850820662379 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated173524819268096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1848306293767 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40261124096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40261124096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 _step63 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed173524793840128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current463 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed168336068448679 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1526 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed26448186 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1850737004032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5628193 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2375954419712 % _timestamp1.7884665059927788e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3540758 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed9168951 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed173524793840128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated5628653 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed166487762154912 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1853253379584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated11455133 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated9169414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed14993059 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1850737004032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated175378072647680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1853253379584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 BS2 - _wandbruntime1475  _runtime14751G{' O YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated176411694352496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed12135296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed12135296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated15915292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1964766797312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _runtime1475.832340427 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated183863347663872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1191 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed178371342878149 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated9719980 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed5966487 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1964766797312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated12135302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated185828114461184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed37797601836544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed28030906 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed9719513 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated37797630934528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated185828114461184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3753029 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1959673953621 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed15895610 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 437549056 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed185825572670976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated183863347663872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed183863322235904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2518544926720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated15915292 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467Af; O RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed12135296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated15915292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 1964766797312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _runtime1475.832340427 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated183863347663872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1191 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed178371342878149 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated9719980 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed5966487 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 1964766797312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated12135302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated185828114461184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed37797601836544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed28030906 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed9719513 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated37797630934528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated185828114461184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3753029 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 1959673953621 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed15895610 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 437549056 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed185825572670976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated183863347663872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed183863322235904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2518544926720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated15915292 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated176411694352496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed12135296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682Bx82 - _wandbruntime1477  _runtime14771c& M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1964525651456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3757358 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1967042027008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed186038838262784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2521450396672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1961946208475 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed40363054675456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated28082852 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1964525651456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated184074338039296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed184074312611328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5973325 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated15933669 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed184074312611328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1967042027008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated176614223062080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated12149183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed176614197634112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime1477.387738838 _step67 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated12149183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak870 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed15913961 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed12149177 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current451 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88025088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated178578683639167 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed9730683 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated186041380066304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated5973773 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed15913961 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated15933669 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed37841663205888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1964460577087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated28082852 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated9731134 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3757361 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated37841692303872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 trainer/global_step849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed28063138 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated184074338039296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2521391469568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 % _timestamp1.7884665873464384e+09D́& M M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated12149183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed176614197634112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime1477.387738838 _step67 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated12149183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak870 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed15913961 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed12149177 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current451 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88025088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated178578683639167 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed9730683 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated186041380066304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated5973773 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed15913961 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated15933669 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed37841663205888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 1964460577087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated28082852 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated9731134 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3757361 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated37841692303872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 trainer/global_step849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed28063138 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated184074338039296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2521391469568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 % _timestamp1.7884665873464384e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 1964525651456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3757358 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 1967042027008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed186038838262784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2521450396672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 1961946208475 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed40363054675456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated28082852 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 1964525651456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated184074338039296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed184074312611328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed5973325 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated15933669 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed184074312611328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 1967042027008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated176614223062080Cv2 - _wandbruntime1477  _runtime1477{(AV  0loss/FBt4/assignment_loss0.3246921778549082 /loss/FRt4/detection_loss2.2079695980422773 /loss/FRt1/assignment_loss1.486030479495445 /loss/FRt4/assignment_loss3.378583863377571 /loss/FRt2/detection_loss2.2615028425266868 /loss/FBt4/detection_loss1.9038209731762226 trainer/global_step849 % _timestamp1.7884665873499088e+09 /loss/FRt2/assignment_loss3.793780074621502 2loss/SRqqt4/assignment_loss1.7042247511300386 1loss/SRqqt2/detection_loss2.1049071568995714 1loss/FBt2/assignment_loss0.18947670679990553 1loss/SRqqt1/detection_loss0.9264464396435548 1loss/SRqqt3/detection_loss2.6881139794392372 _step68 /loss/FRt3/assignment_loss3.984650696768905 2loss/SRqqt2/assignment_loss1.7751167003404011 /loss/FRt1/detection_loss1.9295659961409113 .loss/FRt3/detection_loss2.776019034060565 /loss/FBt2/detection_loss3.3184976100921633 .loss/FBt3/detection_loss3.178015568426677 1loss/SRqqt3/assignment_loss2.253731642418833 1loss/FBt3/assignment_loss0.17762203022305453 0loss/SRqqt4/detection_loss1.906127878848244 /loss/FBt1/detection_loss0.6758635436809415 _runtime 1477.3906876 &loss/total_loss46.076514869639276 epoch1 1loss/SRqqt1/assignment_loss0.972947677008479 1loss/FBt1/assignment_loss0.15880744862317972kaZ  /loss/FRt3/assignment_loss3.984650696768905 2loss/SRqqt2/assignment_loss1.7751167003404011 /loss/FRt1/detection_loss1.9295659961409113 .loss/FRt3/detection_loss2.776019034060565 /loss/FBt2/detection_loss3.3184976100921633 .loss/FBt3/detection_loss3.178015568426677 1loss/SRqqt3/assignment_loss2.253731642418833 1loss/FBt3/assignment_loss0.17762203022305453 0loss/SRqqt4/detection_loss1.906127878848244 /loss/FBt1/detection_loss0.6758635436809415 _runtime 1477.3906876 &loss/total_loss46.076514869639276 epoch1 1loss/SRqqt1/assignment_loss0.972947677008479 1loss/FBt1/assignment_loss0.15880744862317972 0loss/FBt4/assignment_loss0.3246921778549082 /loss/FRt4/detection_loss2.2079695980422773 /loss/FRt1/assignment_loss1.486030479495445 /loss/FRt4/assignment_loss3.378583863377571 /loss/FRt2/detection_loss2.2615028425266868 /loss/FBt4/detection_loss1.9038209731762226 trainer/global_step849 % _timestamp1.7884665873499088e+09 /loss/FRt2/assignment_loss3.793780074621502 2loss/SRqqt4/assignment_loss1.7042247511300386 1loss/SRqqt2/detection_loss2.1049071568995714 1loss/FBt2/assignment_loss0.18947670679990553 1loss/SRqqt1/detection_loss0.9264464396435548 1loss/SRqqt3/detection_loss2.6881139794392372 _step68DqP (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts207.624' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent41.5248 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors01qv (: # proc.memory.rssMB 3165.63671875 disk.sda1.out0 disk.sda.in0 disk.dm-1.out 45.6171875 disk.drbd1260.out 0.20703125 cpu24.675587907702823 proc.cpu.threads17% disk./.usageGB1147.0767669677734 disk.sda1.in0 disk.sda3.in0# disk.nvme0c0n1.in 3391.89453125( disk.nvme0c0n1.out31357.20361328125$ memory_percent94.58398818969727& proc.memory.availableMB 887.359375* proc.memory.percent19.321513175964355) disk./.usagePercent76.98128429272123& disk.nvme0n1.out31357.20361328125# disk.dm-0.out31357.20361328125 disk.sda3.out 45.6171875 disk.sda.out 45.6171875 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 network.sent 1.648479e+06 network.recv507969! disk.nvme0n1.in 3391.89453125 disk.dm-0.in 3391.89453125 disk.drbd1260.in0n (:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts243.093' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent48.618599999999994 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors04x (:  disk.sda2.out0! disk.nvme0n1.in 3391.89453125 disk.dm-0.in 3391.89453125# disk.dm-0.out31459.98095703125 disk.sda3.in0 disk.sda3.out 46.00390625' proc.memory.availableMB 775.1171875* proc.memory.percent19.321513175964355 network.sent 1.724699e+06 network.recv514126) disk./.usagePercent76.97031820999204% disk./.usageGB1146.9133644104004 disk.dm-1.in0 disk.drbd1260.in0& disk.nvme0n1.out31459.98095703125# memory_percent95.2690601348877 disk.sda.in0 disk.sda.out 46.00390625# disk.nvme0c0n1.in 3391.89453125( disk.nvme0c0n1.out31459.98095703125 disk.drbd1260.out 0.2109375 disk.sda1.in0 disk.sda1.out0# proc.memory.rssMB 3165.63671875 disk.sda2.in0 disk.dm-1.out 46.00390625 cpu24.682177696289216 proc.cpu.threads17" (:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts132.554' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent26.5108 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0E_*v (: 襻( proc.memory.availableMB 674.81640625# proc.memory.rssMB 3165.63671875 disk.sda.out 46.44140625 disk.sda2.in0 disk.sda2.out0' disk.nvme0c0n1.out31675.8349609375 disk.dm-1.in0! disk.nvme0n1.in 3391.89453125% disk.nvme0n1.out31675.8349609375" disk.dm-0.out31675.8349609375 network.sent 1.730427e+06 network.recv518373# disk.nvme0c0n1.in 3391.89453125 disk.dm-1.out 46.44140625 disk.drbd1260.out 0.2109375 disk.dm-0.in 3391.89453125 disk.sda1.in0 disk.sda1.out0) disk./.usagePercent76.97034841888109 disk.sda.in0 disk.sda3.in0$ memory_percent95.88124752044678* proc.memory.percent19.321513175964355 cpu24.68887003902397 proc.cpu.threads17% disk./.usageGB1146.9138145446777 disk.drbd1260.in0 disk.sda3.out 46.44140625+ +a (: ϫ͘ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts259.165' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent51.833000000000006 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ef (: ϫԚ% disk.nvme0c0n1.out32022.59765625$ memory_percent96.61712646484375 proc.cpu.threads17 network.sent 1.736114e+06 disk.sda3.out 47.015625 disk.drbd1260.in0# proc.memory.rssMB 3165.63671875# disk.nvme0n1.out32022.59765625 disk.sda.in0 disk.sda2.out0# disk.nvme0c0n1.in 3391.89453125 disk.dm-1.in0 disk.dm-1.out 47.015625 disk.drbd1260.out 0.21484375" proc.memory.availableMB554.25% disk./.usageGB1146.9762153625488! disk.nvme0n1.in 3391.89453125 disk.dm-0.in 3391.89453125 disk.dm-0.out32022.59765625 disk.sda3.in0 disk.sda.out 47.015625* proc.memory.percent19.321513175964355 cpu24.695153356957658 network.recv522522) disk./.usagePercent76.97453619012619 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0X (: ޫ僟 gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts253.331' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent50.666199999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0A+c (: ޫÂ disk.drbd1260.in0# disk.nvme0n1.out32368.77734375 disk.dm-0.in3419.125 disk.sda1.out0$ memory_percent97.37427234649658 disk.sda3.out 47.53515625 disk.sda2.out0 disk.nvme0n1.in3419.125( proc.memory.availableMB 430.19921875# proc.memory.rssMB 3165.63671875* proc.memory.percent19.321513175964355 cpu24.701894565534865 network.recv525945 disk.sda.in0% disk.nvme0c0n1.out32368.77734375 disk.dm-0.out32368.77734375 disk.sda1.in0 proc.cpu.threads17 network.sent 1.741146e+06% disk./.usageGB1147.0908966064453 disk.sda3.in0 disk.sda2.in0 disk.dm-1.in0 disk.dm-1.out 47.53515625 disk.drbd1260.out 0.21484375) disk./.usagePercent76.98223254462805 disk.sda.out 47.53515625 disk.nvme0c0n1.in3419.125_2 - _wandbruntime1555  _runtime1555w  % lr-AdamW/pg10.001169216303774448 % lr-AdamW/pg20.001169216303774448 trainer/global_step899 % _timestamp1.7884666653921585e+09 _runtime1555.433043334 _step69 `V  _step69 % lr-AdamW/pg10.001169216303774448 % lr-AdamW/pg20.001169&q216303774448 trainer/global_step899 % _timestamp1.7884666653921585e+09 _runtime1555.433043334E2 - _wandbruntime1555  _runtime1555b' O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed186538104403728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed196488880089600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed194412841007104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1191 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated6312178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current453 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed42664804573184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3969629 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2076039082496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed6311725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 trainer/global_step899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak870 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2664191112704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2075828236733 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed196488880089600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated16836194 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed10281351 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed40000670303744 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85941248_O' O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated16836194 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated188613958068429 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _step70 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2078555444736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed3969626 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated186538129831696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated10281807 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated194412866435072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated40000699401728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed12829346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56843264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 $ _timestamp1.788466665393127e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2075828236733 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed196488880089600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated16836194 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed10281351 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed40000670303744 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85941248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed186538104403728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed196488880089600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed194412841007104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1191 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated6312178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current453 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed42664804573184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated3969629 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2076039082496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed6311725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 trainer/global_step899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak870 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 2664191112704 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed42711788236800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated29697804 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed12843227 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated196704687484928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed16834863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated12843233 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2078314298880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed29678090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed10292557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed6318599 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed188816219236495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated194623856810496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85927936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated16854571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated194623856810496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated40044760771072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3973961 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated188818759033075 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated186740658541280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime1557.018423436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated16854571 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2667056563712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1191 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2075586123183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed194623831382528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated12843233 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed12843227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed194623831382528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2078314298880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2080830674432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed29678090 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed186740633113312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed16834863 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated6319045 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 trainer/global_step899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979h! M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated3973961 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated188818759033075 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated186740658541280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime1557.018423436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated16854571 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2667056563712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1191 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2O075586123183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed194623831382528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated12843233 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed12843227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed194623831382528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2078314298880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2080830674432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed29678090 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed186740633113312 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed16834863 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated6319045 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 trainer/global_step899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated42711874164736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated10293006 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed3973958 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed40044731673088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated196704687484928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 _step71 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed196702145681408 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed42711788236800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated29697804 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed12843227 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated196704687484928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed16834863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated12843233 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2078314298880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed29678090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed10292557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed6318599 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed188816219236495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated194623856810496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85927936 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated16854571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated194623856810496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated40044760771072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104G2 - _wandbruntime1557  _runtime1557SZ  /loss/FRt3/detection_loss3.5222392535727955 1loss/SRqqt3/detection_loss2.2847563334523815 /loss/FBt1/detection_loss0.6829902666417159 /loss/FBt4/detection_loss2.2800206162035463 %loss/total_loss46.23646747121034 0loss/FBt4/assignment_loss0.2521604160046991 1loss/SRqqt1/detection_loss0.9560063458565209 1loss/SRqqt4/detection_loss1.8174855962657093 _runtime1557.021385973 1loss/SRqqt3/assignment_loss1.974419147796827 /loss/FRt1/detection_loss1.6512911590188741 /loss/FRt3/assignment_loss4.265334584142851 0loss/FRt4/assignment_loss3.8055630356073378 1loss/FBt1/assignment_loss0.09337763747050223 .loss/FRt2/detection_loss2.472745846424784 /loss/FBt2/detection_loss2.8691777306444504 % _timestamp1.7884666669809866e+09 2loss/SRqqt2/assignment_loss1.9972884982995602 0loss/SRqqt4/assignment_loss1.80033219350796 1loss/FBt2/assignment_loss0.18103203119303277 /loss/FRt4/detection_loss2.2759826144576074 epoch1 _step72 0loss/FRt2/assignment_loss3.4835734480903264 2loss/SRqqt1/assignment_loss0.9636374984919611 1loss/FBt3/assignment_loss0.05398458191596616 1loss/SRqqt2/detection_loss2.1294568420826705 .loss/FBt3/detection_loss2.630386219574855 trainer/global_step899 0loss/FRt1/assignment_loss1.7932255744934082VYd^  /loss/FRt4/detection_loss2.2759826144576074 epoch1 _step72 0loss/FRt2/assignment_loss3.4835734480903264 2loss/SRqqt1/assignment_loss0.9636374984919611 1loss/FBt3/assignment_loss0.05398458191596616 1loss/SRqqt2/detection_loss2.1294568420826705 .loss/FBt3/detection_loss2.630386219574855 trainer/global_step899 0loss/FRt1/assignment_loss1.7932255744934082 /loss/FRt3/detection_loss3.5222392535727955 1loss/SRqqt3/detection_loss2.2847563334523815 /loss/FBt1/detection_loss0.6829902666417159 /loss/FBt4/detection_loss2.2800206162035463 %loss/total_loss46.23646747121034 0loss/FBt4/assignment_loss0.2521604160046991 1loss/SRqqt1/detection_loss0.9560063458565209 1loss/SRqqt4/detection_loss1.8174855962657093 _runtime1557.021385973 1loss/SRqqt3/assignment_loss1.974419147796827 /loss/FRt1/detection_loss1.6512911590188741 /loss/FRt3/assignment_loss4.265334584142851 0loss/FRt4/assignment_loss3.8055630356073378 1loss/FBt1/assignment_loss0.09337763747050223 .loss/FRt2/detection_loss2.472745846424784 /loss/FBt2/detection_loss2.8691777306444504 % _timestamp1.7884666669809866e+09 2loss/SRqqt2/assignment_loss1.9972884982995602 0loss/SRqqt4/assignment_loss1.80033219350796 1loss/FBt2/assignment_loss0.18103203119303277Hַw (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts225.766' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.1532 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0V^r (: Ӽ disk.dm-0.in 3538.66015625# disk.nvme0c0n1.in 3538.66015625' disk.nvme0c0n1.out32900.5458984375 disk.dm-1.out 47.9296875 disk.drbd1260.in0# proc.memory.rssMB 3165.63671875 cpu24.707894391063768 proc.cpu.threads17! disk.nvme0n1.in 3538.66015625% disk.nvme0n1.out32900.5458984375" disk.dm-0.out32900.5458984375 disk.sda3.in0 disk.sda3.out 47.9296875 disk.sda2.out0$ memory_percent98.01113605499268( proc.memory.availableMB 325.85546875 network.recv530387% disk./.usageGB1147.2769203186035 disk.sda.out 47.9296875* proc.memory.percent19.321513175964355 network.sent 1.745221e+06) disk./.usagePercent76.99471675203527 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.out0.21875 (:  gpu.0.gpu41.0 gpu.0.memory10+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts321.841' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.3682 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ΩAk (: v% disk./.usageGB1147.5262565612793 disk.sda.in0 disk.dm-1.in0 disk.drbd1260.in0# proc.memory.rssMB 3165.63671875 cpu24.71446023699525 network.recv535766 disk.sda1.in0 disk.sda1.out0 disk.sda.out 48.421875! disk.nvme0n1.in 3538.66015625% disk.nvme0n1.out32959.4287109375 proc.cpu.threads17 disk.sda2.out0 disk.dm-0.in 3538.66015625" disk.dm-0.out32959.4287109375 disk.sda3.in0 disk.sda2.in0# disk.nvme0c0n1.in 3538.66015625 disk.sda3.out 48.421875' disk.nvme0c0n1.out32959.4287109375 disk.dm-1.out 48.421875 disk.drbd1260.out0.21875$ memory_percent98.74606132507324' proc.memory.availableMB 205.4453125* proc.memory.percent19.321513175964355 network.sent 1.833924e+06) disk./.usagePercent77.01144991648829Ғ (: ² gpu.0.gpu76.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts335.845' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.169 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ȍ h (: # proc.memory.rssMB 3165.63671875 disk.sda2.in0 network.sent 1.837765e+06 disk.sda1.out0 disk.dm-1.out48.875 disk.drbd1260.in0( proc.memory.availableMB 108.66796875* proc.memory.percent19.321513175964355 cpu24.721257749062538) disk./.usagePercent77.02825911517898 disk.dm-0.in 3538.66015625 disk.sda2.out0 disk.drbd1260.out 0.22265625 proc.cpu.threads17 network.recv539961$ disk./.usageGB1147.776725769043" disk.dm-0.out33360.7744140625 disk.sda3.out48.875 disk.sda.in0 disk.sda.out48.875 disk.dm-1.in0! disk.nvme0n1.in 3538.66015625 disk.sda1.in0 disk.sda3.in0# disk.nvme0c0n1.in 3538.66015625' disk.nvme0c0n1.out33360.7744140625% disk.nvme0n1.out33360.7744140625$ memory_percent99.336743354797367Ki (:  gpu.0.gpu32.0 gpu.0.memory10+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts327.304' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent65.46079999999999 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 Mk (:  network.recv543539) disk./.usagePercent77.02661580281669 disk.sda.in0 disk.sda.out 49.3515625 disk.sda2.out0# disk.nvme0c0n1.in 3538.66015625 disk.drbd1260.in0 disk.dm-0.in 3538.66015625 network.sent 1.842823e+06 disk.sda2.in0 disk.sda1.in0$ memory_percent99.96185302734375* proc.memory.percent19.321513175964355 cpu24.72724574137874 proc.cpu.threads17$ disk./.usageGB1147.752239227295 disk.sda3.in0 disk.sda3.out 49.3515625 disk.drbd1260.out 0.22265625% disk.nvme0n1.out33741.4658203125 proc.memory.availableMB6.25# proc.memory.rssMB 3165.63671875' disk.nvme0c0n1.out33741.4658203125 disk.dm-1.in0 disk.dm-1.out 49.3515625! disk.nvme0n1.in 3538.66015625" disk.dm-0.out33741.4658203125 disk.sda1.out0`.ߝ (:  gpu.0.gpu65.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp44.0 gpu.0.powerWatts266.388' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent53.27759999999999 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0%h (:  disk.dm-0.in 3705.7265625 disk.sda3.in0 disk.sda3.out 49.734375 disk.sda2.in0# proc.memory.rssMB 3165.63671875) disk./.usagePercent77.02816413638377 disk.dm-1.out 49.734375 disk.nvme0n1.in 3705.7265625$ disk.nvme0n1.out34087.908203125 disk.sda.out 49.734375 disk.sda2.out0& disk.nvme0c0n1.out34087.908203125$ memory_percent99.94900226593018% disk./.usageGB1147.7753105163574 disk.drbd1260.out 0.2265625! disk.dm-0.out34087.908203125 disk.sda1.out0" disk.nvme0c0n1.in 3705.7265625* proc.memory.percent19.321513175964355 cpu24.73344143242747 proc.cpu.threads17 network.sent 1.848795e+06 network.recv547831 disk.dm-1.in0 disk.drbd1260.in0 disk.sda1.in0 disk.sda.in0& proc.memory.availableMB 8.35546875fS (:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp50.0 gpu.0.powerWatts331.038' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.2076 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0,d (: % disk./.usageGB1147.7832984924316 disk.sda2.in0 disk.dm-1.out50.09375 disk.nvme0n1.in 3705.7265625 disk.sda1.out0$ proc.memory.availableMB8.265625 network.sent 1.854462e+06 disk.sda.in0" disk.nvme0c0n1.in 3705.7265625 disk.dm-0.in 3705.7265625 disk.sda1.in0$ memory_percent99.94955062866211# proc.memory.rssMB 3165.63671875 proc.cpu.threads17 disk.sda3.out50.09375 disk.sda.out50.09375* proc.memory.percent19.321513175964355 cpu24.739009511095055 disk.sda3.in0 disk.sda2.out0& disk.nvme0c0n1.out34314.943359375 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.2265625$ disk.nvme0n1.out34314.943359375! disk.dm-0.out34314.943359375 network.recv551921) disk./.usagePercent77.02870021616047/j2 - _wandbruntime1637  _runtime1637}GV  & lr-AdamW/pg10.0011660874027475485 & lr-AdamW/pg20.0011660874027475485 trainer/global_step949 % _timestamp1.7884667476237376e+09 _runtime1637.664226385 _step73  & lr-AdamW/pg10.0011660874027475485 & lr-AdamW/pg20.0011660874027475485 trainer/global_step949 % _timestamp1.7884667476237376e+09 _runtime1637.664226385 _step73Iܶ 2 - _wandbruntime1637  _runtime1637GdO' O TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated6657329 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed17737414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed13523396 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed4186226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed6656865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak870 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated204962385206272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2192344092160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2189827729920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 trainer/global_step949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated17757096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed204962359778304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated17757096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2809903441408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed31260810 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed42203738770944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated196664565310896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated204962385206272 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed207152187508224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 _step74 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680*v  O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed207152187508224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 _step74 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated6657329 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed17737414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed13523396 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed4186226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed6656865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak870 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated204962385206272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2192344092160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2189827729920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 trainer/global_step949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated17757096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed204962359778304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated17757096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 2809903441408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed31260810 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed42203738770944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated196664565310896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated204962385206272 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0JN2 - _wandbruntime1639  _runtime1639 6& M PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed13537277 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated205173375581696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated10854735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated45060691560448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated13537283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 _runtime1639.145511266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4190561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2192102946304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated199058834426983 trainer/global_step949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed17755765 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4190558 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated196867094020480 % _timestamp1.7884667491045256e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed207365453100032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated31312756 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated31312756 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed31293042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated42247829238272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed17755765 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2192102946304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated13537283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed199056294630403 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2189226037891 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2194619321856 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed10854272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1191 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2812862322176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed205173350153728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated17775473 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed42247800140288 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed6663714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2812801297920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated205173375581696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed13537277 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 _step75 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979_\& M OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed6663714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 2812801297920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated205173375581696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed13537277 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 _step75 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed13537277 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated205173375581696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated10854735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated45060691560448 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated13537283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 _runtime1639.145511266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4190561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2192102946304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated199058834426983 trainer/global_step949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed17755765 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4190558 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated196867094020480 % _timestamp1.7884667491045256e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed207365453100032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated31312756 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated31312756 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed31293042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated42247829238272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed17755765 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2192102946304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated13537283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed199056294630403 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2189226037891 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2194619321856 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed10854272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1191 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2812862322176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed205173350153728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated17775473 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed42247800140288 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520Kк2 - _wandbruntime1639  _runtime1639)faZ  2loss/SRqqt1/assignment_loss0.8628356207300686 1loss/SRqqt3/assignment_loss2.075208262182199 /loss/FBt3/detection_loss2.8015790028231486 0loss/FRt3/assignment_loss3.9696845083094354 1loss/SRqqt2/assignment_loss1.522221345808357 .loss/FRt3/detection_loss2.676714056463384 /loss/FBt1/detection_loss0.6333201872585282 .loss/FBt4/detection_loss2.804288474413065 %loss/total_loss45.28412455615652 trainer/global_step949 _step76 2loss/SRqqt4/assignment_loss1.8775118870526244 1loss/FBt2/assignment_loss0.06757983226830569 1loss/FBt4/assignment_loss0.17887494049052924 _runtime1639.147132121 2loss/FBt3/assignment_loss0.009874514029691892 1loss/SRqqt2/detection_loss1.9937363043427467 1loss/SRqqt3/detection_loss2.3562942773867874 0loss/FRt2/assignment_loss3.4640003678562876 0loss/FRt4/assignment_loss3.3313134499270505 0loss/SRqqt1/detection_loss1.167476148306092 epoch1 $ _timestamp1.788466749106824e+09 1loss/FBt1/assignment_loss0.11912386926845211 .loss/FBt2/detection_loss3.212320469892942 /loss/FRt1/detection_loss1.9403371701583914 .loss/FRt4/detection_loss2.419620890659515 1loss/SRqqt4/detection_loss2.1106475373518836 0loss/FRt1/assignment_loss1.4976504956898482 /loss/FRt2/detection_loss2.1919109434871884I`^  /loss/FRt1/detection_loss1.9403371701583914 .loss/FRt4/detection_loss2.419620890659515 1loss/SRqqt4/detection_loss2.1106475373518836 0loss/FRt1/assignment_loss1.4976504956898482 /loss/FRt2/detection_loss2.1919109434871884 2loss/SRqqt1/assignment_loss0.8628356207300686 1loss/SRqqt3/assignment_loss2.075208262182199 /loss/FBt3/detection_loss2.8015790028231486 0loss/FRt3/assignment_loss3.9696845083094354 1loss/SRqqt2/assignment_loss1.522221345808357 .loss/FRt3/detection_loss2.676714056463384 /loss/FBt1/detection_loss0.6333201872585282 .loss/FBt4/detection_loss2.804288474413065 %loss/total_loss45.28412455615652 trainer/global_step949 _step76 2loss/SRqqt4/assignment_loss1.8775118870526244 1loss/FBt2/assignment_loss0.06757983226830569 1loss/FBt4/assignment_loss0.17887494049052924 _runtime1639.147132121 2loss/FBt3/assignment_loss0.009874514029691892 1loss/SRqqt2/detection_loss1.9937363043427467 1loss/SRqqt3/detection_loss2.3562942773867874 0loss/FRt2/assignment_loss3.4640003678562876 0loss/FRt4/assignment_loss3.3313134499270505 0loss/SRqqt1/detection_loss1.167476148306092 epoch1 $ _timestamp1.788466749106824e+09 1loss/FBt1/assignment_loss0.11912386926845211 .loss/FBt2/detection_loss3.212320469892942La (: Ǭ䭢 gpu.0.gpu89.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts245.009' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.001799999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0?i (: Ǭ% disk.nvme0n1.out34597.3330078125" disk.dm-0.out34597.3330078125 disk.sda1.in0 disk.sda3.in0 disk.sda2.in0& proc.memory.availableMB 7.97265625* proc.memory.percent19.321537017822266 disk.drbd1260.in0 disk.drbd1260.out 0.23046875 disk.nvme0n1.in 3705.7265625 disk.sda1.out0 disk.sda3.out 50.5078125 disk.sda.in0' disk.nvme0c0n1.out34597.3330078125 cpu24.7447366333279 network.sent 1.860123e+06 disk.sda2.out0 disk.dm-1.in0! proc.memory.rssMB 3165.640625 network.recv556079# disk./.usageGB1147.78369140625 disk.dm-0.in 3705.7265625 disk.sda.out 50.5078125" disk.nvme0c0n1.in 3705.7265625 disk.dm-1.out 50.5078125$ memory_percent99.95133876800537 proc.cpu.threads17( disk./.usagePercent77.0287265849365֝ (: ֬ gpu.0.gpu72.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts232.425' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent46.48500000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0r_Ii (: ֬ disk.drbd1260.in0% disk.nvme0n1.out34847.4033203125 disk.sda1.in0 disk.sda3.in0 disk.sda.out 51.0078125' disk.nvme0c0n1.out34847.4033203125! proc.memory.rssMB 3165.640625 proc.cpu.threads17 network.sent 1.946689e+06 disk.dm-1.in0 disk.sda1.out0 disk.sda3.out 51.0078125 disk.sda.in0* proc.memory.percent19.321537017822266 cpu24.75049269621891 network.recv561294# disk./.usageGB1147.78369140625 disk.nvme0n1.in 3705.7265625 disk.dm-0.in 3705.7265625" disk.dm-0.out34847.4033203125 disk.sda2.out0$ memory_percent99.94997978210449% proc.memory.availableMB 8.1953125( disk./.usagePercent77.0287265849365 disk.dm-1.out 51.0078125 disk.drbd1260.out 0.23046875 disk.sda2.in0" disk.nvme0c0n1.in 3705.7265625z:Z (:  disk.drbd1260.in0 proc.cpu.threads17 disk.sda.in0! disk.nvme0n1.in 3856.67578125 disk.dm-0.in 3856.67578125 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0& disk.nvme0c0n1.out35396.603515625 disk.drbd1260.out0.234375 network.recv566053$ disk.nvme0n1.out35396.603515625 disk.sda2.in0# disk.nvme0c0n1.in 3856.67578125 disk.dm-1.in0$ memory_percent99.94854927062988% proc.memory.availableMB 8.4296875! proc.memory.rssMB 3165.640625 network.sent 1.952828e+06# disk./.usageGB1147.#jA78369140625! disk.dm-0.out35396.603515625* proc.memory.percent19.321537017822266 cpu24.756083086141306( disk./.usagePercent77.0287265849365 disk.sda3.out 51.55859375 disk.sda.out 51.55859375 disk.sda2.out0 disk.dm-1.out 51.55859375bP- (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts210.234' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent42.0468 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0gKg (: & proc.memory.availableMB 7.91796875 network.sent 1.95844e+06 network.recv569858) disk./.usagePercent77.02715290662353! disk.dm-0.out35742.791015625 disk.sda1.in0% disk./.usageGB1147.7602424621582& disk.nvme0c0n1.out35742.791015625 disk.dm-1.out 51.921875 disk.drbd1260.out0.234375$ disk.nvme0n1.out35742.791015625 disk.sda.in0 disk.sda2.out0$ memory_percent99.95167255401611! disk.nvme0n1.in 3856.67578125 disk.sda1.out0 disk.sda3.in0 disk.sda.out 51.921875! proc.memory.rssMB 3165.640625* proc.memory.percent19.321537017822266 cpu24.76192271817407 proc.cpu.threads17 disk.dm-0.in 3856.67578125# disk.nvme0c0n1.in 3856.67578125 disk.dm-1.in0 disk.drbd1260.in0 disk.sda3.out 51.921875 disk.sda2.in0T&̓ (: 瘛 gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts244.483' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.8966 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0my (:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts233.222' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent46.644400000000005 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ܔt (:  disk.sda3.out 52.30078125 disk.sda.in0% disk./.usageGB1147.9086303710938# disk.dm-0.out35898.91259765625 disk.sda1.out0 disk.sda2.out0# disk.nvme0c0n1.in 3856.67578125( disk.nvme0c0n1.out35898.91259765625) disk./.usagePercent77.03711134369794 disk.sda.out 52.30078125 disk.dm-1.in0 disk.dm-1.out 52.30078125 disk.drbd1260.in0 disk.drbd1260.out 0.23828125$ memory_percent99.94869232177734# proc.memory.availableMB8.40625 disk.sda3.in0 network.sent 1.964128e+06 network.recv574006! disk.nvme0n1.in 3856.67578125 disk.dm-0.in 3856.67578125 disk.sda1.in0 disk.sda2.in0! proc.memory.rssMB 3165.640625& disk.nvme0n1.out35898.91259765625* proc.memory.percent19.321537017822266 cpu24.767267646261157 proc.cpu.threads17,2 - _wandbruntime1720  _runtime1720A  & lr-AdamW/pg10.0011624765706930297 & lr-AdamW/pg20.0011624765706930297 trainer/global_step999 % _timestamp1.7884668308371196e+09 _runtime1720.877794146 _step770)j  & lr-AdamW/pg10.0011624765706930297 & lr-AdamW/pg20.0011624765706930297 trainer/global_step999 % _timestamp1.7884668308371196e+09 _runtime1720.877794146 _step77MoF2 - _wandbruntime1720  _runtime1720V' O D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed215511878549504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2300593697745 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2306132739584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated7003371 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2303616377344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated4402829 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85941248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed206790975362128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2303108066149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed217815494926848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated209094108856245 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed217815494926848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated11406200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed32875762 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 _runtime1720.881263356 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed215511878549504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated18677998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed7002917 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated32895450 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated44406836336128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed4402826 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed47362092914176 _step78 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed47362092914176 _step78 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated4402829 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85941248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed206790975362128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2303108066149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed217815494926848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated209094108856245 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed217815494926848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated11406200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed32875762 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 _runtime1720.881263356 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed215511878549504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144N0@u2 - _wandbruntime1722  _runtime1722_  M D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated218031302322176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 % _timestamp1.7884668323466396e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated18696375 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed209296370024311 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed47409064916480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed18676667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current454 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated206993529499680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step79 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1196 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step999 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed14231327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated32927708 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated215722894352896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed32907994 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4407161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2958253138944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2302865952599 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated215722894352896 OBDeviceStatsMonitor.?oon_train_batch_end/inactive_split.all.allocated11417427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4407158 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed7009812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed11416970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2305380321211 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875b& M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated206993529499680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step79 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1196 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step999 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed14231327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated32927708 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated215722894352896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516375552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed32907994 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4407161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 2958253138944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2302865952599 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated215722894352896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated11417427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4407158 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed7009812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed11416970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2305380321211 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated218031302322176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 437549056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 % _timestamp1.7884668323466396e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated18696375 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed209296370024311 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed47409064916480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed18676667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current454OD7^2 - _wandbruntime1722  _runtime1722 g8W  0loss/FRt1/assignment_loss1.7126418511782373 1loss/FBt3/assignment_loss0.16512841964140534 0loss/SRqqt3/detection_loss2.689352022666557 .loss/FBt2/detection_loss2.769998240470886 trainer/global_step999 1loss/SRqqt4/assignment_loss1.710450044218847 1loss/FBt1/assignment_loss0.23569749789065295 0loss/FBt2/assignment_loss0.2346372385431702 1loss/FBt4/assignment_loss0.15503745486699594 .loss/FRt3/detection_loss3.227686272217677 .loss/FBt1/detection_loss0.719365108147501 epoch1 0loss/FRt3/assignment_loss3.7572372888143244 0loss/SRqqt1/assignment_loss1.16697794481836 /loss/FRt2/detection_loss2.2201791324963174 0loss/FRt2/assignment_loss3.3741456232964993 1loss/SRqqt1/detection_loss0.9827699671740885 /loss/FBt4/detection_loss2.4440200907342575 0loss/SRqqt2/detection_loss2.052877851306124 % _timestamp1.7884668323487804e+09 _step80 0loss/FRt4/assignment_loss3.1128624437052177 /loss/FRt1/detection_loss1.6091817589310398 1loss/SRqqt4/detection_loss2.1666978529476104 _runtime1722.389467789 2loss/SRqqt2/assignment_loss1.5707642441972138 2loss/SRqqt3/assignment_loss1.7330898937641406 /loss/FRt4/detection_loss2.2389361688850125 /loss/FBt3/detection_loss3.2241890063652625 $loss/total_loss45.2739234172774 D[  0loss/FRt4/assignment_loss3.1128624437052177 /loss/FRt1/detection_loss1.6091817589310398 1loss/SRqqt4/detection_loss2.1666978529476104 _runtime1722.389467789 2loss/SRqqt2/assignment_loss1.5707642441972138 2loss/SRqqt3/assignment_loss1.7330898937641406 /loss/FRt4/detection_loss2.2389361688850125 /loss/FBt3/detection_loss3.2241890063652625 $loss/total_loss45.2739234172774 0loss/FRt1/assignment_loss1.7126418511782373 1loss/FBt3/assignment_loss0.16512841964140534 0loss/SRqqt3/detection_loss2.689352022666557 .loss/FBt2/detection_loss2.769998240470886 trainer/global_step999 1loss/SRqqt4/assignment_loss1.710450044218847 1loss/FBt1/assignment_loss0.23569749789065295 0loss/FBt2/assignment_loss0.2346372385431702 1loss/FBt4/assignment_loss0.15503745486699594 .loss/FRt3/detection_loss3.227686272217677 .loss/FBt1/detection_loss0.719365108147501 epoch1 0loss/FRt3/assignment_loss3.7572372888143244 0loss/SRqqt1/assignment_loss1.16697794481836 /loss/FRt2/detection_loss2.2201791324963174 0loss/FRt2/assignment_loss3.3741456232964993 1loss/SRqqt1/detection_loss0.9827699671740885 /loss/FBt4/detection_loss2.4440200907342575 0loss/SRqqt2/detection_loss2.052877851306124 % _timestamp1.7884668323487804e+09 _step80P) (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts234.563' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.9126 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0@r (:  proc.cpu.threads17 network.sent 1.982169e+06 network.recv577526# disk.dm-0.out36245.11181640625 disk.sda1.in0 disk.sda1.out0* proc.memory.percent19.321537017822266 cpu24.772566621412416% disk./.usageGB1147.9086303710938! disk.nvme0n1.in 3856.67578125 disk.dm-0.in 3856.67578125 disk.sda2.in0 disk.sda2.out0( disk.nvme0c0n1.out36245.11181640625 disk.dm-1.out 52.8203125 disk.drbd1260.in0) disk./.usagePercent77.03711134369794 disk.sda3.in0 disk.sda.in0# disk.nvme0c0n1.in 3856.67578125 disk.drbd1260.out 0.23828125# memory_percent99.9504566192627% proc.memory.availableMB 8.1171875! proc.memory.rssMB 3165.640625& disk.nvme0n1.out36245.11181640625 disk.sda3.out 52.8203125 disk.sda.out 52.8203125 disk.dm-1.in0/dN (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts156.816' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent31.363200000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors05 o (: ؞ disk.sda.out 53.2578125( disk.nvme0c0n1.out36457.85986328125 disk.drbd1260.in0$ memory_percent99.95079040527344" proc.memory.availableMB8.0625! proc.memory.rssMB 3165.640625 disk.sda1.in0 disk.sda3.out 53.2578125 disk.dm-1.in0 disk.drbd1260.out 0.2421875* proc.memory.percent19.321537017822266 proc.cpu.threads17 network.recv584052) disk./.usagePercent77.04553475959705 disk.dm-0.in 3995.12109375 disk.sda3.in0 disk.dm-1.out 53.2578125 cpu24.777696003676002 network.sent 2.056534e+06! disk.nvme0n1.in 3995.12109375& disk.nvme0n1.out36457.85986328125# disk.dm-0.out36457.85986328125 disk.sda1.out0 disk.sda2.in0 disk.sda2.out0# disk.nvme0c0n1.in 3995.12109375% disk./.usageGB1148.0341453552246 disk.sda.in0K% (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts252.737' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent50.547399999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0}Cu (: ! proc.memory.rssMB 3165.640625 proc.cpu.threads17$ disk./.usageGB1148.010181427002 disk.sda2.out0 disk.dm-1.out 53.83984375 disk.drbd1260.in0! disk.nvme0n1.in 3995.12109375# disk.dm-0.out36853.18408203125 disk.sda3.out 53.83984375 disk.sda.out 53.83984375 network.sent 2.061186e+06& disk.nvme0n1.out36853.18408203125 disk.dm-0.in 3995.12109375 disk.sda1.in0$ memory_percent99.94924068450928& proc.memory.availableMB 8.31640625* proc.memory.percent19.321537017822266 cpu24.783454555284706 network.recv587234 disk.sda2.in0( disk.nvme0c0n1.out36853.18408203125 disk.dm-1.in0 disk.drbd1260.out 0.2421875 disk.sda1.out0 disk.sda3.in0 disk.sda.in0) disk./.usagePercent77.04392652026695# disk.nvme0c0n1.in 3995.121093756 (:  gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts201.329' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent40.2658 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0hg (: ݎ( disk.nvme0c0n1.out37104.46337890625 disk.sda.out54.125 network.sent 2.067101e+06) disk./.usagePercent77.06232168563226! disk.nvme0n1.in 3995.12109375& disk.nvme0n1.out37104.46337890625 disk.dm-1.out54.125$ memory_percent99.94921684265137 proc.cpu.threads17# disk.dm-0.out37104.46337890625 disk.sda.in0 disk.sda2.in0 disk.sda2.out0# disk.nvme0c0n1.in 3995.12109375 disk.drbd1260.in0 disk.drbd1260.out 0.24609375% proc.memory.availableMB 8.3203125 network.recv591621 disk.dm-0.in 3995.12109375 disk.sda3.in0 disk.sda3.out54.125 disk.dm-1.in0! proc.memory.rssMB 3165.640625* proc.memory.percent19.321537017822266 cpu24.788426031091507% disk./.usageGB1148.2842826843262 disk.sda1.in0 disk.sda1.out07DU+ (: έξ gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts258.125' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent51.625 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0y$w (: έ disk.sda3.out 54.55078125) disk./.usagePercent77.06225461165828 disk.dm-1.out 54.55078125 disk.drbd1260.in0 disk.drbd1260.out 0.24609375 disk.sda2.in0 disk.sda2.out0! proc.memory.rssMB 3165.640625 cpu24.793469250118143# disk.nvme0c0n1.in 3995.12109375 disk.dm-1.in0# disk.dm-0.out37547.96728515625 disk.sda1.in0 disk.sda3.in0& proc.memory.availableMB 8.31640625 proc.cpu.threads17% disk./.usageGB1148.2832832336426( disk.nvme0c0n1.out37547.96728515625 disk.sda1.out0 disk.sda.in0 disk.sda.out 54.55078125$ memory_percent99.94924068450928* proc.memory.percent19.321537017822266 network.sent 2.073923e+06 network.recv595171! disk.nvme0n1.in 3995.12109375& disk.nvme0n1.out37547.96728515625 disk.dm-0.in 3995.12109375#=k (: ݭ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts228.953' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.7906 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0~Ah (: ݭ network.recv599317 disk.dm-0.in 4102.0234375 disk.sda1.in0 disk.sda.in0 disk.sda.out 54.9453125 disk.sda2.in0" disk.nvme0c0n1.in 4102.0234375& proc.memory.availableMB 8.31640625 network.sent 2.078099e+06) disk./.usagePercent77.06225486766581 disk.nvme0n1.in 4102.0234375" disk.dm-0.out38050.2919921875 disk.dm-1.in0 disk.dm-1.out 54.9453125 disk.drbd1260.out0.25 proc.cpu.threads17% disk./.usageGB1148.2832870483398 disk.sda3.in0 disk.sda2.out0$ memory_percent99.94924068450928! proc.memory.rssMB 3165.640625* proc.memory.percent19.321537017822266% disk.nvme0n1.out38050.2919921875 disk.sda1.out0 disk.sda3.out 54.9453125' disk.nvme0c0n1.out38050.2919921875 disk.drbd1260.in0 cpu24.798205362922992~q2 - _wandbruntime1804  _runtime1804@  & lr-AdamW/pg10.0011583868224068724 & lr-AdamW/pg20.0011583868224068724 trainer/global_step1049 % _timestamp1.7884669139743721e+09 _runtime1804.015548589 _step81   _step81 & lr-AdamW/pg10.0011583868224068724 & lr-AdamW/pg20.0011583868224068724 trainer/global_step1049 % _timestamp1.7884669139743721e+09 _runtime1804.015548589Q2 - _wandbruntime1804  _runtime1804 c2]' O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed19579218 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3100851413504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2416747980857 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed228478802345472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed4619426 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3100792473088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 $ _timestamp1.788466913976774e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated34510402 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed219331644453781 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated19598900 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed49710668178432 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2417405024768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed19579218 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated11968234 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed226061397320704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed14911496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed14911496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed228478802345472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak875 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed228478802345472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed4619426 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3100792473088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 $ _timestamp1.788466913976774e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated34510402 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed219331644453781 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated19598900 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed49710668178432 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2417405024768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed19579218 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated11968234 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed226061397320704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed14911496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed14911496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed228478802345472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak875 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed34522946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated34542660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed228692067937280 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current467 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed19597569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated228694609740800 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated14925383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated19617277 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4623761 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2422196616704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _runtime1805.610745982 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed7355218 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated228694609740800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2419020235919 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2416505867307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed11978976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0Bk& M B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated19617277 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated226272413124096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated219538985214799 _step83 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4623758 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated7355682 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 _runtime1805.610745982 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed7355218 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated228694609740800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2419020235919 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2416505867307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed11978976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed34522946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated34542660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed228692067937280 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current467 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed19597569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated228694609740800 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated14925383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated19617277 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated4623761 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2422196616704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed239142109764096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated239144651554304 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated36125354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 trainer/global_step1099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3246165424640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2530387895565 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated239144651554304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated236610941519872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2531193672192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed20500120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 _runtime 1887.22444142 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated236610941519872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed239142109764096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed7693861 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2533710034432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated4836029 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3246224365056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed36105666 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated52059197635584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed15605546 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed236610916091904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed52059109597184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated48812973270528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated36125354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 trainer/global_step1099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3246165424640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2530387895565 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated239144651554304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated236610941519872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2531193672192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed20500120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 _runtime 1887.22444142 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated236610941519872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed239142109764096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed7693861 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2533710034432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated4836029 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88038400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3246224365056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed36105666 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated52059197635584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed15605546 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed236610916091904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed52059109597184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated48812973270528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed239142109764096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated239144651554304 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed15619427 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed36137898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated229779060608707 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed15619427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed229776520812127 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed20518471 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed236821906467328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3249087000064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed227246375030112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1196 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated236821931895296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated20538179 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated36157612 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed12541190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed36137898 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 _step87 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed20518471 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2532660150627 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2535985264128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated36157612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated12541662 trainer/global_step1099 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated227246400458080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed239355375355904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated236821931895296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated7701301 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368)V& M K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated36157612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated12541662 trainer/global_step1099 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated227246400458080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed239355375355904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated236821931895296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated7701301 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed4840358 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated48857034639872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed48857005541888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed15619427 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed36137898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated229779060608707 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed15619427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed229776520812127 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed20518471 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed236821906467328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3249087000064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed227246375030112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1196 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated236821931895296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated20538179 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated36157612 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed12541190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58927104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed36137898 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 _step87 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed20518471 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2532660150627 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2535985264128W42 - _wandbruntime1888  _runtime1888uY  2loss/SRqqt1/assignment_loss0.9854024725485276 %loss/total_loss45.25702417445033 0loss/FRt1/assignment_loss1.4680019186203739 2loss/SRqqt2/assignment_loss1.7950803329891498 /loss/FRt4/detection_loss2.2694488747488886 _step88 /loss/FRt2/assignment_loss3.455955985980698 0loss/FRt3/assignment_loss4.0265185710711355 2loss/SRqqt4/assignment_loss1.7994117533906977 1loss/FBt4/assignment_loss0.15710880465548432 /loss/FRt1/detection_loss1.8137251163372792 1loss/SRqqt3/detection_loss2.2707266468036025 trainer/global_step1099 _runtime 1888.76560695 0loss/FRt4/assignment_loss3.1865686234973727 0loss/FBt3/assignment_loss0.2915274881647747 1loss/SRqqt4/detection_loss1.9953544837363222 /loss/FBt3/detection_loss2.8827382177114487 /loss/FBt4/detection_loss3.2129656699570743 2loss/SRqqt3/assignment_loss1.9724677329576468 1loss/FBt2/assignment_loss0.20569559654104524 .loss/FRt3/detection_loss2.762994904548694 0loss/FBt1/assignment_loss0.1156438393443317 .loss/FRt2/detection_loss2.607931158965147 epoch1 0loss/SRqqt2/detection_loss2.126265710238183 /loss/FBt1/detection_loss0.5698576099659678 -loss/FBt2/detection_loss2.34412349909544 1loss/SRqqt1/detection_loss0.9415091625810449 % _timestamp1.7884669987251174e+09B(]  2loss/SRqqt1/assignment_loss0.9854024725485276 %loss/total_loss45.25702417445033 0loss/FRt1/assignment_loss1.4680019186203739 2loss/SRqqt2/assignment_loss1.7950803329891498 /loss/FRt4/detection_loss2.2694488747488886 _step88 /loss/FRt2/assignment_loss3.455955985980698 0loss/FRt3/assignment_loss4.0265185710711355 2loss/SRqqt4/assignment_loss1.7994117533906977 1loss/FBt4/assignment_loss0.15710880465548432 /loss/FRt1/detection_loss1.8137251163372792 1loss/SRqqt3/detection_loss2.2707266468036025 trainer/global_step1099 _runtime 1888.76560695 0loss/FRt4/assignment_loss3.1865686234973727 0loss/FBt3/assignment_loss0.2915274881647747 1loss/SRqqt4/detection_loss1.9953544837363222 /loss/FBt3/detection_loss2.8827382177114487 /loss/FBt4/detection_loss3.2129656699570743 2loss/SRqqt3/assignment_loss1.9724677329576468 1loss/FBt2/assignment_loss0.20569559654104524 .loss/FRt3/detection_loss2.762994904548694 0loss/FBt1/assignment_loss0.1156438393443317 .loss/FRt2/detection_loss2.607931158965147 epoch1 0loss/SRqqt2/detection_loss2.126265710238183 /loss/FBt1/detection_loss0.5698576099659678 -loss/FBt2/detection_loss2.34412349909544 1loss/SRqqt1/detection_loss0.9415091625810449 % _timestamp1.7884669987251174e+09X/W*n (:  disk.dm-0.in 4269.99609375 disk.sda1.out0 disk.sda.out 57.640625# disk.nvme0c0n1.in 4269.99609375 disk.drbd1260.in0$ memory_percent99.94866847991943" disk.dm-0.out39612.4501953125 disk.sda1.in0 disk.sda2.out0 disk.dm-1.in0& proc.memory.availableMB 8.41015625" proc.memory.rssMB 3165.6484375 proc.cpu.threads17 network.recv624631% disk.nvme0n1.out39612.4501953125 disk.sda.in0* proc.memory.percent19.321584701538086 cpu24.826783419864945 network.sent 2.193005e+06) disk./.usagePercent77.02970555774743 disk.sda3.in0 disk.sda3.out 57.640625 disk.sda2.in0' disk.nvme0c0n1.out39612.4501953125 disk.dm-1.out 57.640625 disk.drbd1260.out 0.26171875% disk./.usageGB1147.7982788085938! disk.nvme0n1.in 4269.99609375y (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts249.806' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.9612 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0~rp (: Ʈߟ# disk.nvme0c0n1.in 4269.99609375 disk.drbd1260.in0* proc.memory.percent19.321584701538086 cpu24.83136642764019 proc.cpu.threads17 disk.sda3.in0% disk./.usageGB1147.6735954284668% disk.nvme0n1.out39806.2822265625" disk.dm-0.out39806.2822265625 disk.sda1.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.out 58.0078125 disk.sda.out 58.0078125' disk.nvme0c0n1.out39806.2822265625$ memory_percent99.94874000549316% proc.memory.availableMB 8.3984375 disk.sda.in0 network.sent 2.279423e+06 network.recv630180) disk./.usagePercent77.02133795149079! disk.nvme0n1.in 4269.99609375 disk.dm-1.in0 disk.drbd1260.out 0.26171875" proc.memory.rssMB 3165.6484375 disk.dm-0.in 4269.99609375 disk.sda1.out0 disk.sda3.out 58.0078125 .ϓ (: Ʈ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts240.976' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.1952 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 (: ծ gpu.0.gpu95.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts228.697' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.7394 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Hoq (: ծ̤ disk.dm-0.in 4269.99609375 disk.sda.in0 disk.sda2.in0$ memory_percent99.95148181915283# proc.memory.rssMB 3165.65234375 network.recv634758 disk.sda1.in0 disk.sda3.out 58.421875 disk.sda.out 58.421875( disk.nvme0c0n1.out40191.53662109375 disk.drbd1260.in0 disk.drbd1260.out0.265625& proc.memory.availableMB 7.94921875& disk.nvme0n1.out40191.53662109375# disk.dm-0.out40191.53662109375 disk.sda1.out0 disk.sda3.in0* proc.memory.percent19.321608543395996 network.sent 2.285497e+06 disk.sda2.out0# disk.nvme0c0n1.in 4269.99609375 disk.dm-1.in0 disk.dm-1.out 58.421875 cpu24.835798616826015 proc.cpu.threads17) disk./.usagePercent77.05066003043254% disk./.usageGB1148.1105155944824! disk.nvme0n1.in 4269.99609375J (: ʍ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts220.122' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent44.0244 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0aib (:  proc.cpu.threads17 network.sent 2.290998e+06 disk.sda.in0 disk.sda.out 58.9453125' disk.nvme0c0n1.out40672.0537109375 disk.dm-1.out 58.9453125% disk.nvme0n1.out40672.0537109375$ memory_percent99.95152950286865& proc.memory.availableMB 7.94140625 network.recv638467) disk./.usagePercent77.05098259992567" disk.dm-0.out40672.0537109375 disk.sda3.out 58.9453125 disk.dm-1.in0 disk.drbd1260.out0.265625# proc.memory.rssMB 3165.65234375* proc.memory.percent19.321608543395996$ disk./.usageGB1148.115322113037 disk.sda1.in0 disk.sda3.in0 disk.drbd1260.in0 disk.dm-0.in 4427.8125 disk.sda1.out0 disk.sda2.in0 disk.sda2.out0 disk.nvme0c0n1.in 4427.8125 disk.nvme0n1.in 4427.8125 cpu24.84054132680644 (: ӽ gpu.0.gpu51.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp41.0 gpu.0.powerWatts116.765' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.352999999999998 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0T`of (:  disk.sda.in0 disk.dm-1.out 59.3359375* proc.memory.percent19.321608543395996$ disk./.usageGB1148.119773864746 disk.sda1.out0 disk.sda2.in0( disk.nvme0c0n1.out40971.46142578125 disk.dm-1.in0& disk.nvme0n1.out40971.46142578125# disk.dm-0.out40971.46142578125$ memory_percent99.95150566101074 network.recv642487 disk.sda3.in0 disk.sda3.out 59.3359375 disk.sda2.out0 disk.drbd1260.in0 disk.drbd1260.out 0.26953125 disk.nvme0n1.in 4427.8125% proc.memory.availableMB 7.9453125 disk.sda1.in0 disk.sda.out 59.3359375 disk.nvme0c0n1.in 4427.8125 disk.dm-0.in 4427.8125# proc.memory.rssMB 3165.65234375 cpu24.84453457274314 proc.cpu.threads17 network.sent 2.29674e+06) disk./.usagePercent77.05128136071814J, (:  gpu.0.gpu98.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts117.939' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.587799999999998 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Kf (:  disk.sda1.out0 disk.sda3.out 59.8984375( disk.nvme0c0n1.out41165.06689453125 disk.dm-0.in 4427.8125$ memory_percent99.95150566101074% proc.memory.availableMB 7.9453125 cpu24.84891236018221 network.recv645907 disk.sda3.in0 disk.dm-1.in0* proc.memory.percent19.321608543395996 proc.cpu.threads17 network.sent 2.30168e+06 disk.sda.in0 disk.sda.out 59.8984375 disk.sda2.out0 disk.nvme0c0n1.in 4427.8125 disk.dm-1.out 59.8984375# disk.dm-0.out41165.06689453125# proc.memory.rssMB 3165.65234375) disk./.usagePercent77.05128136071814$ disk./.usageGB1148.119773864746 disk.sda1.in0 disk.sda2.in0 disk.drbd1260.in0 disk.drbd1260.out 0.26953125 disk.nvme0n1.in 4427.8125& disk.nvme0n1.out41165.06689453125o2 - _wandbruntime1969  _runtime1969n   & lr-AdamW/pg10.0011487846327828434 & lr-AdamW/pg20.0011487846327828434 trainer/global_step1149 % _timestamp1.7884670796055474e+09 _runtime1969.646524158 _step89  _runtime1969.646524158 _step89 & lr-AdamW/pg10.0011487846327828434 & lr-AdamW/pg20.0011487846327828434 trainer/global_step1149 % _timestamp1.7884670796055474e+09Y2 - _wandbruntime1969  _runtime19699 O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated16299602 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed21421022 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3391900459008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2644982319616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed249805417182720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2641513441869 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 trainer/global_step1149 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed54407913098752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed13092736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5052629 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3391959399424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed237170281799728 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2644982319616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak875 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed239811795241597 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed16299596 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated239814335037969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current459 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8040110 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5052626 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated8040566 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2644027810273 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 437549056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673BP ' O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3391900459008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2644982319616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed249805417182720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2641513441869 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 trainer/global_step1149 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed54407913098752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed13092736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5052629 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3391959399424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed237170281799728 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2644982319616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak875 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed239811795241597 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed16299596 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated239814335037969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current459 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8040110 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5052626 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated8040566 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2644027810273 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 437549056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated16299602 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed21421022 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240Zam92 - _wandbruntime1971  _runtime1971S& M JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated237372835937280 _runtime1971.606059777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed240016596206035 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed13103960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed37752850 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed16313477 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23979 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed5056958 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2649773911552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2647257536000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated37772564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated21459081 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed250018682774528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed237372810509312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated54454966486528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2649773911552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3394804452352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated37772564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated21459081 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak875 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed250018682774528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed237372810509312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated54454966486528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2649773911552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3394804452352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed240016596206035 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed13103960[k2 - _wandbruntime1971  _runtime1971b X  0loss/FRt4/assignment_loss2.7850318651450308 2loss/SRqqt2/assignment_loss1.9263481104079587 1loss/SRqqt3/assignment_loss1.513725677977747 .loss/FRt2/detection_loss2.256404187653091 /loss/FBt1/detection_loss0.7882871603714067 &loss/total_loss43.924387767260896 .loss/FRt3/detection_loss2.479534632364909 0loss/FRt2/assignment_loss3.6584860990335653 /loss/FRt1/detection_loss1.5550673195125104 /loss/FBt2/detection_loss3.3178235217928886 0loss/FRt1/assignment_loss1.6456011740060954 0loss/FRt3/assignment_loss3.2458518727620445 2loss/SRqqt4/assignment_loss1.8110356330683437 1loss/FBt4/assignment_loss0.23976808873174543 /loss/FRt4/detection_loss2.2432512073140396 .loss/FBt3/detection_loss2.787462830543518 epoch1 0loss/FBt3/assignment_loss0.2394156678346917 1loss/SRqqt4/detection_loss1.9822193805617516 /loss/FBt4/detection_loss2.3450755178928375 .loss/FBt2/assignment_loss0.29066518379841 0loss/SRqqt1/detection_loss1.054573751090802 0loss/SRqqt2/detection_loss2.122477583863117 1loss/SRqqt3/detection_loss2.4541204166946127 $ _timestamp1.788467081566635e+09 trainer/global_step1149 2loss/SRqqt1/assignment_loss1.0246081413360955 1loss/FBt1/assignment_loss0.15755274350368392 _runtime1971.607005861 _step92~j\  1loss/SRqqt3/detection_loss2.4541204166946127 $ _timestamp1.788467081566635e+09 trainer/global_step1149 2loss/SRqqt1/assignment_loss1.0246081413360955 1loss/FBt1/assignment_loss0.15755274350368392 _runtime1971.607005861 _step92 0loss/FRt4/assignment_loss2.7850318651450308 2loss/SRqqt2/assignment_loss1.9263481104079587 1loss/SRqqt3/assignment_loss1.513725677977747 .loss/FRt2/detection_loss2.256404187653091 /loss/FBt1/detection_loss0.7882871603714067 &loss/total_loss43.924387767260896 .loss/FRt3/detection_loss2.479534632364909 0loss/FRt2/assignment_loss3.6584860990335653 /loss/FRt1/detection_loss1.5550673195125104 /loss/FBt2/detection_loss3.3178235217928886 0loss/FRt1/assignment_loss1.6456011740060954 0loss/FRt3/assignment_loss3.2458518727620445 2loss/SRqqt4/assignment_loss1.8110356330683437 1loss/FBt4/assignment_loss0.23976808873174543 /loss/FRt4/detection_loss2.2432512073140396 .loss/FBt3/detection_loss2.787462830543518 epoch1 0loss/FBt3/assignment_loss0.2394156678346917 1loss/SRqqt4/detection_loss1.9822193805617516 /loss/FBt4/detection_loss2.3450755178928375 .loss/FBt2/assignment_loss0.29066518379841 0loss/SRqqt1/detection_loss1.054573751090802 0loss/SRqqt2/detection_loss2.122477583863117\yV"& (: ဲ gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp40.0 gpu.0.powerWatts116.069' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.213800000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0RDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed250051870635505 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 % _timestamp1.7884671626932824e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2758770967040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed247296717278928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2761287329280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated16993652 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated260471266391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed260468724601344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated8386104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated247296742706896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed56756085250560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated257709979062272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8385647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2761287329280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2757667724981 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed257709953634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2755153356577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed260468724601344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated13655333 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 _runtime2052.734073484 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed13654873 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 _step94 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed22341924 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated16993652 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated53219110204928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3537004143616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed257709953634304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed22341924 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5269229 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed16993646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5269226 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated56756173288960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed53219081106944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527a' O RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed257709953634304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed22341924 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5269229 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed16993646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5269226 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated56756173288960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed53219081106944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 trainer/global_step1199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed39335570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current460 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated257709979062272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3537063084032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated260471266391552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2758770967040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated22361606 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated250054410431877 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current19688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed250051870635505 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736368 % _timestamp1.7884671626932824e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2758770967040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed247296717278928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2761287329280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated16993652 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated260471266391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed260468724601344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated8386104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated247296742706896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed56756085250560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated257709979062272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8385647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2761287329280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58940416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796372 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2757667724981 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed257709953634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2755153356577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed260468724601344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated13655333 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 _runtime2052.734073484 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed13654873 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 _step94 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed22341924 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23979 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated16993652 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated53219110204928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3537004143616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248^-g2 - _wandbruntime2054  _runtime20544Uf M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current472 _step95 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated260684531996672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed247499245988512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated39387516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed13665878 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90122240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed250256671599943 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed22360275 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated8392789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated17007533 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed257920944009728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2757425611431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61024256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated53263171574272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed39367802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2763562558976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated257920969437696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated250259211396523 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated39387516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 3539963153408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 _runtime2054.223527384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated17007533 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2761046183424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current469 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed53263142476288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated13666350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3539902129152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed22360275 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated22379983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed260681990193152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3P $;& M ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 2759939980043 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated39387516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed13665878 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90122240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed250256671599943 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed22360275 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated8392789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541803520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated17007533 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796580 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed257920944009728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2757425611431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61024256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19714 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated53263171574272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed39367802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2763562558976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated257920969437696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19708 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated250259211396523 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated39387516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 3539963153408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 _runtime2054.223527384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated17007533 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2761046183424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current469 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed53263142476288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated13666350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3539902129152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed22360275 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated22379983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed260681990193152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current472 _step95 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated260684531996672 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541803520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011457 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19708 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed247499245988512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736368 F&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 *jet/accuracy_2_of_50.3892956078052521 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.5833333134651184 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2920617461204529 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 ;"Purity/3FRt*SRqqt1FBt/event_purity0.022727273404598236 ("Purity/1FRt*SRqqt4FBt/event_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 ("Purity/*FRt4SRqqt1FBt/event_purity0 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 & Purity/*FRt3SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.2819116413593292 8 Purity/1FRt*SRqqt2FBt/FRt_purity0.43923038244247437 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.35555556416511536 & Purity/2FRt2SRqqt*FBt/FBt_purity1 +jet/accuracy_2_of_40.30911150574684143 /"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.234375 ("Purity/0FRt1SRqqt3FBt/event_purity0 7 Purity/*FRt3SRqqt1FBt/FRt_purity0.3277777433395386 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity1 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.30893900990486145 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.3017857074737549 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 :"Purity/*FRt*SRqqt*FBt/event_purity0.20614536106586456 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.3007313907146454 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 +jet/accuracy_3_of_50.08800634741783142 particle/accuracy_2_of_51 + Purity/0FRt0SRqqt4FBt/FBt_purity0.8125 ("Purity/0FRt3SRqqt0FBt/event_purity0 ("Purity/4FRt2SRqqt*FBt/event_purity0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity1 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ("Purity/0FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9976958632469177 ("Purity/*FRt4SRqqt0FBt/event_purity0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 particle/accuracy_3_of_61 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2291666716337204 ;"Purity/0FRt2SRqqt1FBt/event_purity0.019969278946518898 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.31052225828170776 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.33778801560401917 8 Purity/2FRt1SRqqt*FBt/FRt_purity0.29832160472869873 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.3888888657093048 ("Purity/2FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt*SRqqt2FBt/FRt_purity0.4314191937446594 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity1 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 :"Purity/*FRt*SRqqt1FBt/event_purity0.40159887075424194 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.42105263471603394 particle/accuracy_2_of_41 particle/accuracy_3_of_71 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.29163840413093567 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/0FRt2SRqqt4FBct/event_proportion0 jet/accuracy_4_of_60 7"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.40476194024086 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9908666014671326 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 * Purity/*FRt*SRqqt4FBt/FBt_purity0.875 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9983022212982178 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ("Purity/3FRt1SRqqt*FBt/event_purity0 particle/accuracy_5_of_51 jet/accuracy_6_of_60 ("Purity/*FRt3SRqqt2FBt/event_purity0 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9965590834617615 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 7 Purity/3FRt*SRqqt*FBt/FRt_purity0.2669013738632202 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 jet/accuracy_4_of_70 particle/accuracy_6_of_71 ("Purity/4FRt1SRqqt*FBt/event_purity0 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.32367151975631714 9"Purity/0FRt*SRqqt1FBt/event_purity0.6423121094703674 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9986966252326965 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 ;"Purity/2FRt0SRqqt*FBt/event_purity0.027722440659999847 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 ("Purity/3FRt2SRqqt*FBt/event_purity0 particle/accuracy_2_of_61 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 8 Purity/*FRt2SRqqt1FBt/FRt_purity0.31674695014953613 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.306775838136673 *jet/accuracy_1_of_10.3880918025970459 *jet/accuracy_1_of_50.9249321222305298 ("Purity/3FRt4SRqqt*FBt/event_purity0 particle/accuracy_7_of_70.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity0.8333333730697632 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 <"Purity/2FRt*SRqqt0FBt/event_purity0.0076773627661168575 ("Purity/2FRt3SRqqt*FBt/event_purity0 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 ("Purity/1FRt2SRqqt2FBt/event_purity0 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.31362414360046387 ("Purity/2FRt0SRqqt2FBt/event_purity0 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 particle/accuracy_1_of_31 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.2857142984867096 ("Purity/4FRt2SRqqt0FBt/event_purity0 :"Purity/0FRt0SRqqt3FBt/event_purity0.15000000596046448 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9946236610412598 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ;"Purity/0FRt2SRqqt0FBt/event_purity0.013329767622053623 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 particle/accuracy_4_of_51 9"Purity/*FRt1SRqqt1FBt/event_purity0.1028757244348526 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9903023838996887 ("Purity/*FRt1SRqqt3FBt/event_purity0 7"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.40476194024086 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.9166666865348816 _runtime2119.876565458 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 9"Purity/*FRt0SRqqt1FBt/event_purity0.6751497387886047 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 *jet/accuracy_1_of_20.6904533505439758 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.3461538553237915 ("Purity/3FRt*SRqqt2FBt/event_purity0 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9983022212982178 ("Purity/*FRt3SRqqt*FBt/event_purity0 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity0 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9878411889076233 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 & Purity/1FRt1SRqqt4FBt/FRt_purity0 trainer/global_step1207 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9867706894874573 & Purity/0FRt2SRqqt1FBt/FBt_purity1 :"Purity/1FRt*SRqqt0FBt/event_purity0.09701564908027649 + Purity/1FRt*SRqqt3FBt/FRt_purity0.4375 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.7142857313156128 7 Purity/1FRt*SRqqt0FBt/FRt_purity0.2701546549797058 ;"Purity/2FRt*SRqqt*FBt/event_purity0.009491385892033577 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 particle/accuracy_4_of_71 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 ("Purity/2FRt4SRqqt*FBt/event_purity0 :"Purity/*FRt*SRqqt2FBt/event_purity0.20258015394210815 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.33739736676216125 particle/accuracy_3_of_51 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.30706891417503357 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.46666666865348816 :"Purity/1FRt1SRqqt1FBt/event_purity0.03828616812825203 ("Purity/1FRt2SRqqt*FBt/event_purity0 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity1 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.22375309467315674 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 6 Purity/2FRt*SRqqt0FBt/FRt_purity0.278270423412323 ;"Purity/3FRt0SRqqt*FBt/event_purity0.022727273404598236 & Purity/0FRt3SRqqt*FBt/FBt_purity1 & Purity/3FRt0SRqqt2FBt/FBt_purity1 epoch1 :"Purity/*FRt*SRqqt3FBt/event_purity0.08045977354049683 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 ("Purity/*FRt4SRqqt*FBt/event_purity0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.341269850730896 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9967741966247559 5validation_average_jet_accuracy0.376395046710968 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9980030655860901 ("Purity/1FRt0SRqqt3FBt/event_purity0 ("Purity/2FRt2SRqqt1FBt/event_purity0 6 Purity/2FRt3SRqqt0FBt/FRt_purity0.343722939491272 particle/accuracy_1_of_71 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.26492971181869507 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.9074074625968933 5 Purity/1FRt1SRqqt*FBt/FBt_purity0.99237459897995 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 ("Purity/3FRt1SRqqt1FBt/event_purity0 8 Purity/1FRt1SRqqt1FBt/FRt_purity0.27659085392951965 jet/accuracy_5_of_50 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 6 Purity/*FRt0SRqqt2FBt/FBt_purity0.988223671913147 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.29056665301322937 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9744444489479065 ("Purity/2FRt1SRqqt2FBt/event_purity0 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.17973856627941132 7 Purity/*FRt1SRqqt1FBt/FRt_purity0.2905005216598511 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.4187609851360321 ("Purity/1FRt3SRqqt1FBt/event_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.3888888955116272 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 ("Purity/3FRt4SRqqt0FBt/event_purity0 & Purity/3FRt*SRqqt2FBt/FBt_purity1 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 particle/accuracy_3_of_31 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 ("Purity/3FRt3SRqqt*FBt/event_purity0 ("Purity/3FRt3SRqqt0FBt/event_purity0 6 Purity/*FRt1SRqqt*FBt/FBt_purity0.991884171962738 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9989594221115112 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.2857142984867096 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 6 Purity/3FRt2SRqqt*FBt/FRt_purity0.319444477558136 particle/accuracy_4_of_41 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.28465622663497925 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.2575661540031433 particle/accuracy_1_of_11 ("Purity/2FRt4SRqqt0FBt/event_purity0 & Purity/*FRt3SRqqt1FBt/FBt_purity1 & Purity/2FRt2SRqqt1FBt/FBt_purity1 jet/accuracy_2_of_60.5 particle/accuracy_1_of_61 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.4564647376537323 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.3205128014087677 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 particle/accuracy_1_of_41 ("Purity/2FRt3SRqqt1FBt/event_purity0 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.44117647409439087 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 :"Purity/1FRt*SRqqt*FBt/event_purity0.10442212224006653 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.3333333432674408 8 Purity/*FRt*SRqqt0FBt/FRt_purity0.27103355526924133 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.2826032042503357 & Purity/3FRt1SRqqt*FBt/FBt_purity1 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 7 Purity/2FRt1SRqqt0FBt/FRt_purity0.2823125422000885 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.3888888657093048 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9866157174110413 9"Purity/*FRt0SRqqt3FBt/event_purity0.0922619104385376 :"Purity/*FRt2SRqqt0FBt/event_purity0.00538600841537118 9"Purity/0FRt*SRqqt2FBt/event_purity0.2934178113937378 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.45103684067726135 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.3888888657093048 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity0 jet/accuracy_7_of_70 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 *jet/accuracy_1_of_40.8876534700393677 5 Purity/*FRt1SRqqt3FBt/FBt_purity0.89682537317276 & Purity/*FRt1SRqqt4FBt/FRt_purity0 :"Purity/0FRt*SRqqt0FBt/event_purity0.20165058970451355 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.6052631735801697 ("Purity/*FRt0SRqqt4FBt/event_purity0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity1 )loss/val_total_loss43.35549122483641 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9919427633285522 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.3031727075576782 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.3938831686973572 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt2SRqqt0FBt/event_purity0 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.19027777016162872 +jet/accuracy_2_of_30.20266474783420563 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 (particle/accuracy0.8442016243934631 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.25 6 Purity/1FRt2SRqqt2FBt/FRt_purity0.236111119389534 particle/accuracy_5_of_71 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity0 ) Purity/4FRt1SRqqt0FBt/FRt_purity0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32387542724609375 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.28115832805633545 *jet/accuracy_1_of_30.8237913846969604 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity0 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.28973814845085144 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.31668585538864136 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 ("Purity/4FRt*SRqqt0FBt/event_purity0 & Purity/4FRt2SRqqt*FBt/FRt_purity0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 ("Purity/1FRt3SRqqt*FBt/event_purity0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity0 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.3496376872062683 ("Purity/2FRt0SRqqt3FBt/event_purity0 7 Purity/*FRt1SRqqt0FBt/FRt_purity0.2997017204761505 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.39418843388557434 +jet/accuracy_3_of_60.07017543911933899 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.36532261967658997 ("Purity/1FRt2SRqqt1FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 ("Purity/*FRt*SRqqt4FBt/event_purity0 jet/accuracy_3_of_70 jet/accuracy_6_of_70 ) Purity/*FRt1SRqqt3FBt/FRt_purity0.25 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/event_purity0 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.34500178694725037 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 +jet/accuracy_3_of_30.01996731385588646 8 Purity/*FRt2SRqqt2FBt/FRt_purity0.25641027092933655 ("Purity/4FRt0SRqqt*FBt/event_purity0 9"Purity/*FRt0SRqqt2FBt/event_purity0.2779350280761719 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.9220306873321533 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.30052927136421204 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 & Purity/3FRt*SRqqt*FBt/FBt_purity1 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 ("Purity/*FRt1SRqqt4FBt/event_purity0 6 Purity/*FRt1SRqqt2FBt/FBt_purity0.979268491268158 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.30076804757118225 ("Purity/2FRt3SRqqt0FBt/event_purity0 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 7 Purity/3FRt*SRqqt1FBt/FRt_purity0.3141414225101471 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ( Purity/*FRt*SRqqt4FBt/FRt_purity0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.3999760150909424 8"Purity/0FRt0SRqqt1FBt/event_purity0.999193549156189 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.42491498589515686 ,particle/sensitivity0.27086567878723145 particle/accuracy_4_of_61 ("Purity/1FRt4SRqqt*FBt/event_purity0 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9815475344657898 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.5777778029441833 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt0FBt/event_purity0 ( Purity/1FRt*SRqqt4FBt/FRt_purity0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity0.35597869753837585 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 ;"Purity/1FRt1SRqqt0FBt/event_purity0.009883998893201351 8 Purity/2FRt*SRqqt1FBt/FRt_purity0.33027520775794983 +jet/accuracy_3_of_40.05031299963593483 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity0.21875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity1 & Purity/4FRt0SRqqt*FBt/FRt_purity0 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9915336966514587 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9936347007751465 & Purity/0FRt4SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.2932221293449402 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.9479166865348816 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.28595152497291565 & Purity/*FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt2SRqqt1FBt/event_purity0 particle/accuracy_6_of_61 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.9311111569404602 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 & Purity/3FRt0SRqqt*FBt/FBt_purity1 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt3SRqqt1FBt/FRt_purity0.15000000596046448 :"Purity/1FRt0SRqqt1FBt/event_purity0.23779621720314026 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 ("Purity/3FRt2SRqqt0FBt/event_purity0 particle/accuracy_2_of_31 jet/accuracy_1_of_61 ("Purity/0FRt3SRqqt1FBt/event_purity0 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.31447717547416687 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.3263888657093048 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity1 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 8 Purity/3FRt0SRqqt1FBt/FRt_purity0.38461539149284363 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 & Purity/2FRt3SRqqt*FBt/FBt_purity1 ) Purity/1FRt1SRqqt3FBt/FRt_purity0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 7 Purity/*FRt3SRqqt*FBt/FRt_purity0.2684131860733032 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.32318899035453796 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 9"Purity/0FRt0SRqqt*FBt/event_purity0.8279340863227844 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9897587895393372 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.29897215962409973 :"Purity/2FRt0SRqqt1FBt/event_purity0.05111111328005791 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.43988099694252014 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 _step97 7 Purity/1FRt0SRqqt2FBt/FRt_purity0.4750438332557678 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.3283718526363373 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.3141108453273773 ;"Purity/*FRt2SRqqt1FBt/event_purity0.008995036594569683 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.2937735617160797 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.4016534090042114 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4113408327102661 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.24235792458057404 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 7 Purity/1FRt0SRqqt*FBt/FRt_purity0.2795655429363251 ;"Purity/1FRt1SRqqt*FBt/event_purity0.016641614958643913 ("Purity/1FRt3SRqqt0FBt/event_purity0 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.281962513923645 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 6 Purity/2FRt4SRqqt*FBt/FRt_purity0.699999988079071 & Purity/3FRt2SRqqt1FBt/FBt_purity1 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9855339527130127 7 Purity/1FRt*SRqqt1FBt/FRt_purity0.3232732117176056 & Purity/*FRt0SRqqt4FBt/FRt_purity1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity0 & Purity/1FRt3SRqqt2FBt/FRt_purity1 7 Purity/1FRt*SRqqt*FBt/FBt_purity0.9938427805900574 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.3852209150791168 particle/accuracy_2_of_21 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.2617630660533905 particle/accuracy_5_of_61 7 Purity/1FRt4SRqqt0FBt/FRt_purity0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity1 * Purity/4FRt*SRqqt*FBt/FRt_purity0.125 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.9311508536338806 :"Purity/1FRt0SRqqt0FBt/event_purity0.22233738005161285 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.26893743872642517 -jet/accuracy_4_of_40.0011947430903092027 & Purity/1FRt0SRqqt4FBt/FBt_purity1 & Purity/2FRt*SRqqt3FBt/FBt_purity1 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.4080306887626648 ("Purity/0FRt2SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.2868228256702423 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.317750483751297 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 & Purity/3FRt2SRqqt*FBt/FBt_purity1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.32111677527427673 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.40416669845581055 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 +jet/accuracy_4_of_50.01856154575943947 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.2222222238779068 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.29783549904823303 6 Purity/0FRt0SRqqt1FBt/FBt_purity0.999193549156189 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 & Purity/0FRt2SRqqt2FBt/FBt_purity1 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.23333331942558289 6 Purity/1FRt*SRqqt1FBt/FBt_purity0.998289942741394 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 (particle/f_score0.35188841819763184 jet/accuracy_5_of_60 jet/accuracy_5_of_70 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.6228070259094238 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 & Purity/*FRt2SRqqt2FBt/FBt_purity1 & Purity/*FRt1SRqqt4FBt/FBt_purity1 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.30238619446754456 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.2580110728740692 ("Purity/2FRt1SRqqt1FBt/event_purity0 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 <"Purity/*FRt2SRqqt*FBt/event_purity0.0062639894895255566 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ("Purity/2FRt*SRqqt3FBt/event_purity0 6 Purity/*FRt0SRqqt1FBt/FBt_purity0.999199628829956 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 & Purity/2FRt*SRqqt1FBt/FBt_purity1 & Purity/0FRt4SRqqt1FBt/FBt_purity1 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.2916666567325592 ("Purity/4FRt1SRqqt0FBt/event_purity0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.2720632255077362 ("Purity/0FRt0SRqqt0FBt/event_purity1 & Purity/*FRt4SRqqt*FBt/FBt_purity1 ("Purity/0FRt*SRqqt4FBt/event_purity0 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.28559625148773193 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 & Purity/4FRt2SRqqt0FBt/FRt_purity0 % _timestamp1.7884672298335736e+09 9"Purity/0FRt1SRqqt1FBt/event_purity0.1710294932126999 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9764814972877502 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.4037037193775177 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 particle/accuracy_1_of_51 ("Purity/0FRt2SRqqt2FBt/event_purity0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 ("Purity/2FRt1SRqqt*FBt/event_purity0 & Purity/1FRt2SRqqt2FBt/FBt_purity1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.43312519788742065 ;"Purity/2FRt0SRqqt0FBt/event_purity0.025046324357390404 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 & Purity/2FRt3SRqqt1FBt/FBt_purity1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.2793620228767395 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 & Purity/3FRt0SRqqt1FBt/FBt_purity1 8 Purity/3FRt1SRqqt*FBt/FRt_purity0.22119343280792236 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.2916666567325592 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.31372538208961487 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_1_of_70.5 ) Purity/4FRt1SRqqt*FBt/FRt_purity0.25 ("Purity/*FRt3SRqqt1FBt/event_purity0 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 ("Purity/3FRt*SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 particle/accuracy_3_of_41 ("Purity/*FRt2SRqqt2FBt/event_purity0 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity0.23129001259803772 ("Purity/3FRt0SRqqt2FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FBt_purity1 & Purity/2FRt2SRqqt2FBt/FBt_purity1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ;"Purity/*FRt1SRqqt2FBt/event_purity0.014784946106374264 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.22233738005161285 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 ("Purity/*FRt3SRqqt0FBt/event_purity0 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.4642857015132904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.4405086040496826 & Purity/1FRt3SRqqt2FBt/FBt_purity1 9"Purity/0FRt1SRqqt0FBt/event_purity0.2580110728740692 9"Purity/0FRt2SRqqt*FBt/event_purity0.0153876431286335 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9907549619674683 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.25341784954071045 ("Purity/2FRt*SRqqt2FBt/event_purity0 6 Purity/2FRt4SRqqt0FBt/FRt_purity0.699999988079071 7 Purity/1FRt4SRqqt*FBt/FRt_purity0.1666666716337204 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity0 * Purity/4FRt*SRqqt0FBt/FRt_purity0.125 9"Purity/*FRt0SRqqt*FBt/event_purity0.3987656235694885 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.28686535358428955 :"Purity/0FRt0SRqqt2FBt/event_purity0.38842737674713135 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.32788535952568054 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.5 + Purity/0FRt*SRqqt4FBt/FBt_purity0.8125 ("Purity/0FRt4SRqqt*FBt/event_purity0 9"Purity/*FRt0SRqqt0FBt/event_purity0.1889907270669937 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.39851850271224976 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 :"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.38005954027175903 & Purity/3FRt*SRqqt1FBt/FBt_purity1 +particle/specificity0.9503277540206909 ("Purity/4FRt*SRqqt*FBt/event_purity0 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 7 Purity/1FRt1SRqqt*FBt/FRt_purity0.3076096177101135 ("Purity/*FRt2SRqqt3FBt/event_purity0 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/0FRt3SRqqt*FBt/event_purity0 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9986265301704407 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 ("Purity/1FRt1SRqqt4FBt/event_purity0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity1 :"Purity/*FRt1SRqqt0FBt/event_purity0.13592588901519775 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.4279528558254242 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.3259774148464203 & Purity/1FRt1SRqqt4FBt/FBt_purity1 :"Purity/*FRt*SRqqt0FBt/event_purity0.12197820097208023 :"Purity/*FRt1SRqqt*FBt/event_purity0.12302157282829285 :"Purity/0FRt*SRqqt*FBt/event_purity0.36814287304878235 :"Purity/3FRt0SRqqt1FBt/event_purity0.07692307978868484 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 ("Purity/1FRt4SRqqt0FBt/event_purity0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 ("Purity/2FRt2SRqqt*FBt/event_purity0 ("Purity/2FRt2SRqqt0FBt/event_purity0 particle/accuracy_1_of_21 :"Purity/0FRt1SRqqt*FBt/event_purity0.22658193111419678 8 Purity/1FRt1SRqqt2FBt/FRt_purity0.42517009377479553 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 & Purity/4FRt0SRqqt0FBt/FRt_purity0 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 +jet/accuracy_2_of_20.09662239998579025 ("Purity/4FRt0SRqqt0FBt/event_purity0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.35348644852638245 9"Purity/1FRt*SRqqt1FBt/event_purity0.1339573860168457 ;"Purity/3FRt*SRqqt*FBt/event_purity0.004608294926583767 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 7 Purity/*FRt0SRqqt2FBt/FRt_purity0.4560329020023346 ("Purity/1FRt*SRqqt3FBt/event_purity0 8 Purity/2FRt*SRqqt2FBt/FRt_purity0.43638885021209717 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 7 Purity/3FRt0SRqqt*FBt/FRt_purity0.2727273106575012 ("Purity/3FRt0SRqqt0FBt/event_purity0 & Purity/3FRt1SRqqt1FBt/FBt_purity1 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 :"Purity/0FRt*SRqqt3FBt/event_purity0.11419752985239029 :"Purity/1FRt*SRqqt2FBt/event_purity0.08221534639596939 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.33039000630378723 6 Purity/2FRt3SRqqt*FBt/FRt_purity0.322857141494751 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.26230159401893616 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_2_of_71 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity0.25 :"Purity/1FRt0SRqqt2FBt/event_purity0.10796213895082474 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9523809552192688 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 7 Purity/*FRt0SRqqt4FBt/FBt_purity0.8500000238418579 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.2919066548347473 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 9"Purity/1FRt1SRqqt2FBt/event_purity0.0535714291036129 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 ;"Purity/2FRt*SRqqt1FBt/event_purity0.024449564516544342 jet/accuracy_2_of_70 ("Purity/1FRt0SRqqt4FBt/event_purity0 ,&Purity/0FRt4SRqqt4FBt/event_proYportion0 :"Purity/1FRt0SRqqt*FBt/event_purity0.21650436520576477 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 8"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.699999988079071 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.5 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.5p~  9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.3141108453273773 ;"Purity/*FRt2SRqqt1FBt/event_purity0.008995036594569683 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.2937735617160797 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.4016534090042114 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4113408327102661 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.24235792458057404 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 7 Purity/1FRt0SRqqt*FBt/FRt_purity0.2795655429363251 ;"Purity/1FRt1SRqqt*FBt/event_purity0.016641614958643913 ("Purity/1FRt3SRqqt0FBt/event_purity0 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.281962513923645 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 6 Purity/2FRt4SRqqt*FBt/FRt_purity0.699999988079071 & Purity/3FRt2SRqqt1FBt/FBt_purity1 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9855339527130127 7 Purity/1FRt*SRqqt1FBt/FRt_purity0.3232732117176056 & Purity/*FRt0SRqqt4FBt/FRt_purity1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity0 & Purity/1FRt3SRqqt2FBt/FRt_purity1 7 Purity/1FRt*SRqqt*FBt/FBt_purity0.9938427805900574 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.3852209150791168 particle/accuracy_2_of_21 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.2617630660533905 particle/accuracy_5_of_61 7 Purity/1FRt4SRqqt0FBt/FRt_purity0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity1 * Purity/4FRt*SRqqt*FBt/FRt_purity0.125 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.9311508536338806 :"Purity/1FRt0SRqqt0FBt/event_purity0.22233738005161285 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.26893743872642517 -jet/accuracy_4_of_40.0011947430903092027 & Purity/1FRt0SRqqt4FBt/FBt_purity1 & Purity/2FRt*SRqqt3FBt/FBt_purity1 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.4080306887626648 ("Purity/0FRt2SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.2868228256702423 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.317750483751297 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 & Purity/3FRt2SRqqt*FBt/FBt_purity1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.32111677527427673 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.40416669845581055 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 +jet/accuracy_4_of_50.01856154575943947 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.2222222238779068 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.29783549904823303 6 Purity/0FRt0SRqqt1FBt/FBt_purity0.999193549156189 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 & Purity/0FRt2SRqqt2FBt/FBt_purity1 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.23333331942558289 6 Purity/1FRt*SRqqt1FBt/FBt_purity0.998289942741394 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 (particle/f_score0.35188841819763184 jet/accuracy_5_of_60 jet/accuracy_5_of_70 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.6228070259094238 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 & Purity/*FRt2SRqqt2FBt/FBt_purity1 & Purity/*FRt1SRqqt4FBt/FBt_purity1 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.30238619446754456 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.2580110728740692 ("Purity/2FRt1SRqqt1FBt/event_purity0 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 <"Purity/*FRt2SRqqt*FBt/event_purity0.0062639894895255566 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ("Purity/2FRt*SRqqt3FBt/event_purity0 6 Purity/*FRt0SRqqt1FBt/FBt_purity0.999199628829956 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 & Purity/2FRt*SRqqt1FBt/FBt_purity1 & Purity/0FRt4SRqqt1FBt/FBt_purity1 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.2916666567325592 ("Purity/4FRt1SRqqt0FBt/event_purity0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.2720632255077362 ("Purity/0FRt0SRqqt0FBt/event_purity1 & Purity/*FRt4SRqqt*FBt/FBt_purity1 ("Purity/0FRt*SRqqt4FBt/event_purity0 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.28559625148773193 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 & Purity/4FRt2SRqqt0FBt/FRt_purity0 % _timestamp1.7884672298335736e+09 9"Purity/0FRt1SRqqt1FBt/event_purity0.1710294932126999 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9764814972877502 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.4037037193775177 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 particle/accuracy_1_of_51 ("Purity/0FRt2SRqqt2FBt/event_purity0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 ("Purity/2FRt1SRqqt*FBt/event_purity0 & Purity/1FRt2SRqqt2FBt/FBt_purity1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.43312519788742065 ;"Purity/2FRt0SRqqt0FBt/event_purity0.025046324357390404 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 & Purity/2FRt3SRqqt1FBt/FBt_purity1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.2793620228767395 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 & Purity/3FRt0SRqqt1FBt/FBt_purity1 8 Purity/3FRt1SRqqt*FBt/FRt_purity0.22119343280792236 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.2916666567325592 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.31372538208961487 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_1_of_70.5 ) Purity/4FRt1SRqqt*FBt/FRt_purity0.25 ("Purity/*FRt3SRqqt1FBt/event_purity0 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 ("Purity/3FRt*SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 particle/accuracy_3_of_41 ("Purity/*FRt2SRqqt2FBt/event_purity0 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity0.23129001259803772 ("Purity/3FRt0SRqqt2FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FBt_purity1 & Purity/2FRt2SRqqt2FBt/FBt_purity1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ;"Purity/*FRt1SRqqt2FBt/event_purity0.014784946106374264 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.22233738005161285 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 ("Purity/*FRt3SRqqt0FBt/event_purity0 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.4642857015132904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.4405086040496826 & Purity/1FRt3SRqqt2FBt/FBt_purity1 9"Purity/0FRt1SRqqt0FBt/event_purity0.2580110728740692 9"Purity/0FRt2SRqqt*FBt/event_purity0.0153876431286335 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9907549619674683 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.25341784954071045 ("Purity/2FRt*SRqqt2FBt/event_purity0 6 Purity/2FRt4SRqqt0FBt/FRt_purity0.699999988079071 7 Purity/1FRt4SRqqt*FBt/FRt_purity0.1666666716337204 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity0 * Purity/4FRt*SRqqt0FBt/FRt_purity0.125 9"Purity/*FRt0SRqqt*FBt/event_purity0.3987656235694885 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.28686535358428955 :"Purity/0FRt0SRqqt2FBt/event_purity0.38842737674713135 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.32788535952568054 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.5 + Purity/0FRt*SRqqt4FBt/FBt_purity0.8125 ("Purity/0FRt4SRqqt*FBt/event_purity0 9"Purity/*FRt0SRqqt0FBt/event_purity0.1889907270669937 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.39851850271224976 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 :"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.38005954027175903 & Purity/3FRt*SRqqt1FBt/FBt_purity1 +particle/specificity0.9503277540206909 ("Purity/4FRt*SRqqt*FBt/event_purity0 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 7 Purity/1FRt1SRqqt*FBt/FRt_purity0.3076096177101135 ("Purity/*FRt2SRqqt3FBt/event_purity0 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/0FRt3SRqqt*FBt/event_purity0 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9986265301704407 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 ("Purity/1FRt1SRqqt4FBt/event_purity0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity1 :"Purity/*FRt1SRqqt0FBt/event_purity0.13592588901519775 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.4279528558254242 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.3259774148464203 & Purity/1FRt1SRqqt4FBt/FBt_purity1 :"Purity/*FRt*SRqqt0FBt/event_purity0.12197820097208023 :"Purity/*FRt1SRqqt*FBt/event_purity0.12302157282829285 :"Purity/0FRt*SRqqt*FBt/event_purity0.36814287304878235 :"Purity/3FRt0SRqqt1FBt/event_purity0.07692307978868484 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 ("Purity/1FRt4SRqqt0FBt/event_purity0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 ("Purity/2FRt2SRqqt*FBt/event_purity0 ("Purity/2FRt2SRqqt0FBt/event_purity0 particle/accuracy_1_of_21 :"Purity/0FRt1SRqqt*FBt/event_purity0.22658193111419678 8 Purity/1FRt1SRqqt2FBt/FRt_purity0.42517009377479553 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 & Purity/4FRt0SRqqt0FBt/FRt_purity0 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 +jet/accuracy_2_of_20.09662239998579025 ("Purity/4FRt0SRqqt0FBt/event_purity0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.35348644852638245 9"Purity/1FRt*SRqqt1FBt/event_purity0.1339573860168457 ;"Purity/3FRt*SRqqt*FBt/event_purity0.004608294926583767 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 7 Purity/*FRt0SRqqt2FBt/FRt_purity0.4560329020023346 ("Purity/1FRt*SRqqt3FBt/event_purity0 8 Purity/2FRt*SRqqt2FBt/FRt_purity0.43638885021209717 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 7 Purity/3FRt0SRqqt*FBt/FRt_purity0.2727273106575012 ("Purity/3FRt0SRqqt0FBt/event_purity0 & Purity/3FRt1SRqqt1FBt/FBt_purity1 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 :"Purity/0FRt*SRqqt3FBt/event_purity0.11419752985239029 :"Purity/1FRt*SRqqt2FBt/event_purity0.08221534639596939 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.33039000630378723 6 Purity/2FRt3SRqqt*FBt/FRt_purity0.322857141494751 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.26230159401893616 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_2_of_71 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity0.25 :"Purity/1FRt0SRqqt2FBt/event_purity0.10796213895082474 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9523809552192688 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 7 Purity/*FRt0SRqqt4FBt/FBt_purity0.8500000238418579 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.2919066548347473 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 9"Purity/1FRt1SRqqt2FBt/event_purity0.0535714291036129 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 ;"Purity/2FRt*SRqqt1FBt/event_purity0.024449564516544342 jet/accuracy_2_of_70 ("Purity/1FRt0SRqqt4FBt/event_purity0 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 :"Purity/1FRt0SRqqt*FBt/event_purity0.21650436520576477 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 8"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.699999988079071 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.5 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.5 7 Purity/*FRt0SRqqt*FBt/FRt_purity0.2774779796600342 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 7 Purity/2FRt*SRqqt*FBt/FRt_purity0.2957284152507782 & Purity/1FRt0SRqqt4FBt/FRt_purity1 & Purity/2FRt0SRqqt3FBt/FBt_purity1 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 & Purity/1FRt*SRqqt4FBt/FBt_purity1 7 Purity/*FRt1SRqqt1FBt/FBt_purity0.9981929659843445 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9943577647209167 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 7 Purity/2FRt0SRqqt1FBt/FRt_purity0.3043915629386902 ("Purity/2FRt1SRqqt0FBt/event_purity0 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 7 Purity/1FRt0SRqqt1FBt/FRt_purity0.3632417917251587 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9903225302696228 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 *jet/accuracy_2_of_50.3892956078052521 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.5833333134651184 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2920617461204529 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 ;"Purity/3FRt*SRqqt1FBt/event_purity0.022727273404598236 ("Purity/1FRt*SRqqt4FBt/event_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 ("Purity/*FRt4SRqqt1FBt/event_purity0 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 & Purity/*FRt3SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.2819116413593292 8 Purity/1FRt*SRqqt2FBt/FRt_purity0.43923038244247437 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.35555556416511536 & Purity/2FRt2SRqqt*FBt/FBt_purity1 +jet/accuracy_2_of_40.30911150574684143 /"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.234375 ("Purity/0FRt1SRqqt3FBt/event_purity0 7 Purity/*FRt3SRqqt1FBt/FRt_purity0.3277777433395386 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity1 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.30893900990486145 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.3017857074737549 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 :"Purity/*FRt*SRqqt*FBt/event_purity0.20614536106586456 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.3007313907146454 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 +jet/accuracy_3_of_50.08800634741783142 particle/accuracy_2_of_51 + Purity/0FRt0SRqqt4FBt/FBt_purity0.8125 ("Purity/0FRt3SRqqt0FBt/event_purity0 ("Purity/4FRt2SRqqt*FBt/event_purity0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity1 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ("Purity/0FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9976958632469177 ("Purity/*FRt4SRqqt0FBt/event_purity0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 particle/accuracy_3_of_61 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2291666716337204 ;"Purity/0FRt2SRqqt1FBt/event_purity0.019969278946518898 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.31052225828170776 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.33778801560401917 8 Purity/2FRt1SRqqt*FBt/FRt_purity0.29832160472869873 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.3888888657093048 ("Purity/2FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt*SRqqt2FBt/FRt_purity0.4314191937446594 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity1 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 :"Purity/*FRt*SRqqt1FBt/event_purity0.40159887075424194 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.42105263471603394 particle/accuracy_2_of_41 particle/accuracy_3_of_71 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.29163840413093567 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 jet/accuracy_4_of_60 7"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.40476194024086 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9908666014671326 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 * Purity/*FRt*SRqqt4FBt/FBt_purity0.875 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9983022212982178 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ("Purity/3FRt1SRqqt*FBt/event_purity0 particle/accuracy_5_of_51 jet/accuracy_6_of_60 ("Purity/*FRt3SRqqt2FBt/event_purity0 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9965590834617615 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 7 Purity/3FRt*SRqqt*FBt/FRt_purity0.2669013738632202 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 jet/accuracy_4_of_70 particle/accuracy_6_of_71 ("Purity/4FRt1SRqqt*FBt/event_purity0 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.32367151975631714 9"Purity/0FRt*SRqqt1FBt/event_purity0.6423121094703674 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9986966252326965 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 ;"Purity/2FRt0SRqqt*FBt/event_purity0.027722440659999847 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 ("Purity/3FRt2SRqqt*FBt/event_purity0 particle/accuracy_2_of_61 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 8 Purity/*FRt2SRqqt1FBt/FRt_purity0.31674695014953613 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.306775838136673 *jet/accuracy_1_of_10.3880918025970459 *jet/accuracy_1_of_50.9249321222305298 ("Purity/3FRt4SRqqt*FBt/event_purity0 particle/accuracy_7_of_70.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity0.8333333730697632 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 <"Purity/2FRt*SRqqt0FBt/event_purity0.0076773627661168575 ("Purity/2FRt3SRqqt*FBt/event_purity0 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 ("Purity/1FRt2SRqqt2FBt/event_purity0 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.31362414360046387 ("Purity/2FRt0SRqqt2FBt/event_purity0 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 particle/accuracy_1_of_31 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.2857142984867096 ("Purity/4FRt2SRqqt0FBt/event_purity0 :"Purity/0FRt0SRqqt3FBt/event_purity0.15000000596046448 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9946236610412598 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ;"Purity/0FRt2SRqqt0FBt/event_purity0.013329767622053623 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 particle/accuracy_4_of_51 9"Purity/*FRt1SRqqt1FBt/event_purity0.1028757244348526 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9903023838996887 ("Purity/*FRt1SRqqt3FBt/event_purity0 7"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.40476194024086 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.9166666865348816 _runtime2119.876565458 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 9"Purity/*FRt0SRqqt1FBt/event_purity0.6751497387886047 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 *jet/accuracy_1_of_20.6904533505439758 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.3461538553237915 ("Purity/3FRt*SRqqt2FBt/event_purity0 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9983022212982178 ("Purity/*FRt3SRqqt*FBt/event_purity0 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity0 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9878411889076233 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 & Purity/1FRt1SRqqt4FBt/FRt_purity0 trainer/global_step1207 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9867706894874573 & Purity/0FRt2SRqqt1FBt/FBt_purity1 :"Purity/1FRt*SRqqt0FBt/event_purity0.09701564908027649 + Purity/1FRt*SRqqt3FBt/FRt_purity0.4375 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.7142857313156128 7 Purity/1FRt*SRqqt0FBt/FRt_purity0.2701546549797058 ;"Purity/2FRt*SRqqt*FBt/event_purity0.009491385892033577 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 particle/accuracy_4_of_71 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 ("Purity/2FRt4SRqqt*FBt/event_purity0 :"Purity/*FRt*SRqqt2FBt/event_purity0.20258015394210815 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.33739736676216125 particle/accuracy_3_of_51 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.30706891417503357 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.46666666865348816 :"Purity/1FRt1SRqqt1FBt/event_purity0.03828616812825203 ("Purity/1FRt2SRqqt*FBt/event_purity0 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity1 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.22375309467315674 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 6 Purity/2FRt*SRqqt0FBt/FRt_purity0.278270423412323 ;"Purity/3FRt0SRqqt*FBt/event_purity0.022727273404598236 & Purity/0FRt3SRqqt*FBt/FBt_purity1 & Purity/3FRt0SRqqt2FBt/FBt_purity1 epoch1 :"Purity/*FRt*SRqqt3FBt/event_purity0.08045977354049683 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 ("Purity/*FRt4SRqqt*FBt/event_purity0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.341269850730896 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9967741966247559 5validation_average_jet_accuracy0.376395046710968 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9980030655860901 ("Purity/1FRt0SRqqt3FBt/event_purity0 ("Purity/2FRt2SRqqt1FBt/event_purity0 6 Purity/2FRt3SRqqt0FBt/FRt_purity0.343722939491272 particle/accuracy_1_of_71 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.26492971181869507 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.9074074625968933 5 Purity/1FRt1SRqqt*FBt/FBt_purity0.99237459897995 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 ("Purity/3FRt1SRqqt1FBt/event_purity0 8 Purity/1FRt1SRqqt1FBt/FRt_purity0.27659085392951965 jet/accuracy_5_of_50 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 6 Purity/*FRt0SRqqt2FBt/FBt_purity0.988223671913147 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.29056665301322937 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9744444489479065 ("Purity/2FRt1SRqqt2FBt/event_purity0 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.17973856627941132 7 Purity/*FRt1SRqqt1FBt/FRt_purity0.2905005216598511 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.4187609851360321 ("Purity/1FRt3SRqqt1FBt/event_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.3888888955116272 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 ("Purity/3FRt4SRqqt0FBt/event_purity0 & Purity/3FRt*SRqqt2FBt/FBt_purity1 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 particle/accuracy_3_of_31 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 ("Purity/3FRt3SRqqt*FBt/event_purity0 ("Purity/3FRt3SRqqt0FBt/event_purity0 6 Purity/*FRt1SRqqt*FBt/FBt_purity0.991884171962738 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9989594221115112 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.2857142984867096 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 6 Purity/3FRt2SRqqt*FBt/FRt_purity0.319444477558136 particle/accuracy_4_of_41 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.28465622663497925 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.2575661540031433 particle/accuracy_1_of_11 ("Purity/2FRt4SRqqt0FBt/event_purity0 & Purity/*FRt3SRqqt1FBt/FBt_purity1 & Purity/2FRt2SRqqt1FBt/FBt_purity1 jet/accuracy_2_of_60.5 particle/accuracy_1_of_61 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.4564647376537323 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.3205128014087677 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 particle/accuracy_1_of_41 ("Purity/2FRt3SRqqt1FBt/event_purity0 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.44117647409439087 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 :"Purity/1FRt*SRqqt*FBt/event_purity0.10442212224006653 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.3333333432674408 8 Purity/*FRt*SRqqt0FBt/FRt_purity0.27103355526924133 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.2826032042503357 & Purity/3FRt1SRqqt*FBt/FBt_purity1 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 7 Purity/2FRt1SRqqt0FBt/FRt_purity0.2823125422000885 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.3888888657093048 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9866157174110413 9"Purity/*FRt0SRqqt3FBt/event_purity0.0922619104385376 :"Purity/*FRt2SRqqt0FBt/event_purity0.00538600841537118 9"Purity/0FRt*SRqqt2FBt/event_purity0.2934178113937378 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.45103684067726135 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.3888888657093048 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity0 jet/accuracy_7_of_70 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 *jet/accuracy_1_of_40.8876534700393677 5 Purity/*FRt1SRqqt3FBt/FBt_purity0.89682537317276 & Purity/*FRt1SRqqt4FBt/FRt_purity0 :"Purity/0FRt*SRqqt0FBt/event_purity0.20165058970451355 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.6052631735801697 ("Purity/*FRt0SRqqt4FBt/event_purity0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity1 )loss/val_total_loss43.35549122483641 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9919427633285522 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.3031727075576782 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.3938831686973572 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt2SRqqt0FBt/event_purity0 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.19027777016162872 +jet/accuracy_2_of_30.20266474783420563 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 (particle/accuracy0.8442016243934631 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.25 6 Purity/1FRt2SRqqt2FBt/FRt_purity0.236111119389534 particle/accuracy_5_of_71 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity0 ) Purity/4FRt1SRqqt0FBt/FRt_purity0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32387542724609375 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.28115832805633545 *jet/accuracy_1_of_30.8237913846969604 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity0 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.28973814845085144 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.31668585538864136 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 ("Purity/4FRt*SRqqt0FBt/event_purity0 & Purity/4FRt2SRqqt*FBt/FRt_purity0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 ("Purity/1FRt3SRqqt*FBt/event_purity0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity0 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.3496376872062683 ("Purity/2FRt0SRqqt3FBt/event_purity0 7 Purity/*FRt1SRqqt0FBt/FRt_purity0.2997017204761505 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.39418843388557434 +jet/accuracy_3_of_60.07017543911933899 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.36532261967658997 ("Purity/1FRt2SRqqt1FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 ("Purity/*FRt*SRqqt4FBt/event_purity0 jet/accuracy_3_of_70 jet/accuracy_6_of_70 ) Purity/*FRt1SRqqt3FBt/FRt_purity0.25 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/event_purity0 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 :"Purity/2FRt1SRq<7qt*FBt/SRqqt_purity0.34500178694725037 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 +jet/accuracy_3_of_30.01996731385588646 8 Purity/*FRt2SRqqt2FBt/FRt_purity0.25641027092933655 ("Purity/4FRt0SRqqt*FBt/event_purity0 9"Purity/*FRt0SRqqt2FBt/event_purity0.2779350280761719 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.9220306873321533 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.30052927136421204 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 & Purity/3FRt*SRqqt*FBt/FBt_purity1 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 ("Purity/*FRt1SRqqt4FBt/event_purity0 6 Purity/*FRt1SRqqt2FBt/FBt_purity0.979268491268158 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.30076804757118225 ("Purity/2FRt3SRqqt0FBt/event_purity0 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 7 Purity/3FRt*SRqqt1FBt/FRt_purity0.3141414225101471 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ( Purity/*FRt*SRqqt4FBt/FRt_purity0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.3999760150909424 8"Purity/0FRt0SRqqt1FBt/event_purity0.999193549156189 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.42491498589515686 ,particle/sensitivity0.27086567878723145 particle/accuracy_4_of_61 ("Purity/1FRt4SRqqt*FBt/event_purity0 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9815475344657898 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.5777778029441833 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt0FBt/event_purity0 ( Purity/1FRt*SRqqt4FBt/FRt_purity0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity0.35597869753837585 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 ;"Purity/1FRt1SRqqt0FBt/event_purity0.009883998893201351 8 Purity/2FRt*SRqqt1FBt/FRt_purity0.33027520775794983 +jet/accuracy_3_of_40.05031299963593483 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity0.21875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity1 & Purity/4FRt0SRqqt*FBt/FRt_purity0 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9915336966514587 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9936347007751465 & Purity/0FRt4SRqqt*FBt/FBt_purity1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.2932221293449402 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.9479166865348816 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.28595152497291565 & Purity/*FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt2SRqqt1FBt/event_purity0 particle/accuracy_6_of_61 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.9311111569404602 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 & Purity/3FRt0SRqqt*FBt/FBt_purity1 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt3SRqqt1FBt/FRt_purity0.15000000596046448 :"Purity/1FRt0SRqqt1FBt/event_purity0.23779621720314026 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 ("Purity/3FRt2SRqqt0FBt/event_purity0 particle/accuracy_2_of_31 jet/accuracy_1_of_61 ("Purity/0FRt3SRqqt1FBt/event_purity0 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.31447717547416687 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.3263888657093048 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity1 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 8 Purity/3FRt0SRqqt1FBt/FRt_purity0.38461539149284363 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 & Purity/2FRt3SRqqt*FBt/FBt_purity1 ) Purity/1FRt1SRqqt3FBt/FRt_purity0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 7 Purity/*FRt3SRqqt*FBt/FRt_purity0.2684131860733032 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.32318899035453796 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 9"Purity/0FRt0SRqqt*FBt/event_purity0.8279340863227844 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9897587895393372 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.29897215962409973 :"Purity/2FRt0SRqqt1FBt/event_purity0.05111111328005791 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.43988099694252014 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 _step97 7 Purity/1FRt0SRqqt2FBt/FRt_purity0.4750438332557678 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.3283718526363373 ,&Purity/3FRt1SRqqt3FBt/event_proportion0aD` (:  disk.drbd1260.in0 disk.drbd1260.out0.296875 disk.sda1.in0 disk.sda3.in0$ memory_percent77.87284851074219 network.sent 2.525773e+06) disk./.usagePercent77.28616980942881 disk.nvme0n1.out 47771.28125 disk.sda3.out 64.76171875 disk.sda2.in0* proc.memory.percent19.072532653808594% disk./.usageGB1151.6197814941406 disk.dm-1.in0! disk.nvme0n1.in 4984.06640625 disk.dm-0.out 47771.28125 disk.sda1.out0 disk.sda.in0 disk.sda.out 64.76171875 cpu25.00691727514999 network.recv694522# disk.nvme0c0n1.in 4984.06640625 disk.dm-1.out 64.76171875 disk.dm-0.in 4984.06640625 disk.sda2.out0% proc.memory.availableMB 3625.3125 proc.memory.rssMB 3124.84375 proc.cpu.threads17" disk.nvme0c0n1.out 47771.28125' (:  gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp33.0 gpu.0.powerWatts94.158' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent18.8316 gpu.0.smClock1275 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0>l{ (: Ǵ gpu.0.gpu84.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp44.0 gpu.0.powerWatts315.707' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent63.141400000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3712951353856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated23526865 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed272718706561024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated264517524541611 trainer/global_step1249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed18016792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed23507182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5548913 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated55548242740736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 _step99 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2903958580224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated18016798 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated272718731988992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed272718706561024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated275622690569216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed275620148778496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated272718731988992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2901442217472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated23526865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3713008196608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed23507182 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8890594 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2900143503067 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed59261164996608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2901442217472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated14439963 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated41543663 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated18016798 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5548910 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated59261250937344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2897629134659 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 % _timestamp1.7884673090375092e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2903958580224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current459 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed14439504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed264514984745235 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed18016792E6' O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated18016798 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5548910 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated59261250937344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 2897629134659 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 % _timestamp1.7884673090375092e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 2903958580224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current459 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed14439504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed264514984745235 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed18016792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated8891050 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed55548213642752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 3712951353856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated23526865 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed272718706561024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated264517524541611 trainer/global_step1249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed18016792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed23507182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5548913 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated55548242740736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 _step99 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 2903958580224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated18016798 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated272718731988992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed272718706561024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated275622690569216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed275620148778496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated272718731988992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 2901442217472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated23526865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 3713008196608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed23507182 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed8890594 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 2900143503067 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed59261164996608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 2901442217472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated14439963 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated41543663car\2 - _wandbruntime2200  _runtime2200淕\ M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated18030679 WCDel]ٟ&viceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed272929696936448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2903717433856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed55592275012096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed59308140802048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3715865789952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated5553245 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed23525533 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated8898003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed264719785709673 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2899901389513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed41556206 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2903717433856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated23545242 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated23545242 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated275835956174336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2906233809920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 trainer/global_step1249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated41575921 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed5553242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated272929722364416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current468 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated272929722364416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated55592304110080 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed8897538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated41575921 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated59308228826624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 _runtime2200.599425776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2906233809920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated275835956174336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed272929696936448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed14450780 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated264722325506257 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated261819909748128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 _step100 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated18030679 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146dH& M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated41575921 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated59308228826624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 _runtime2200.599425776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 2906233809920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated275835956174336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed272929696936448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed14450780 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated264722325506257 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated261819909748128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 _step100 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated18030679 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated18030679 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed272929696936448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 2903717433856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed55592275012096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed59308140802048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 3715865789952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated5553245 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed23525533 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated8898003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed264719785709673 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 2899901389513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed41556206 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 2903717433856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated23545242 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated23545242 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated275835956174336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 2906233809920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 trainer/global_step1249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated41575921 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed5553242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated272929722364416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current468 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated272929722364416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated55592304110080 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed8897538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2dI2 - _wandbruntime2200  _runtime2200OSWU  /loss/FRt1/detection_loss1.8857701998637166 2loss/SRqqt3/assignment_loss1.9555505193406488 2loss/SRqqt4/assignment_loss1.7820872325593462 0loss/FRt4/assignment_loss2.8087604217072752 0loss/FBt2/assignment_loss1.6378330318534258 _runtime2200.601474299 _step101 0loss/FRt1/assignment_loss1.4285997706790303 1loss/SRqqt2/detection_loss2.0437043278023252 $ _timestamp1.788467310561101e+09 1loss/SRqqt2/assignment_loss2.048653479925422 /loss/FRt2/assignment_loss3.668356844748574 /loss/FRt3/assignment_loss3.115030294491185 1loss/FBt1/assignment_loss0.09223904733817824 0loss/FBt4/assignment_loss0.0945895829286706 .loss/FBt2/detection_loss2.744627989255465 0loss/FBt3/assignment_loss0.2919867044858014 /loss/FRt3/detection_loss2.6110864323046474 .loss/FRt4/detection_loss2.329509635237937 .loss/FBt3/detection_loss2.714235512415568 epoch2 trainer/global_step1249 /loss/FRt2/detection_loss2.2717913002118295 1loss/SRqqt1/assignment_loss1.129244066851814 /loss/FBt1/detection_loss0.7605827901372973 .loss/FBt4/detection_loss2.151511321465174 #loss/total_loss44.930200434016 1loss/SRqqt1/detection_loss1.1547707192415049 1loss/SRqqt3/detection_loss2.2318992745585557 1loss/SRqqt4/detection_loss1.9777799346126044^X5Y  .loss/FRt4/detection_loss2.329509635237937 .loss/FBt3/detection_loss2.714235512415568 epoch2 trainer/global_step1249 /loss/FRt2/detection_loss2.2717913002118295 1loss/SRqqt1/assignment_loss1.129244066851814 /loss/FBt1/detection_loss0.7605827901372973 .loss/FBt4/detection_loss2.151511321465174 #loss/total_loss44.930200434016 1loss/SRqqt1/detection_loss1.1547707192415049 1loss/SRqqt3/detection_loss2.2318992745585557 1loss/SRqqt4/detection_loss1.9777799346126044 /loss/FRt1/detection_loss1.8857701998637166 2loss/SRqqt3/assignment_loss1.9555505193406488 2loss/SRqqt4/assignment_loss1.7820872325593462 0loss/FRt4/assignment_loss2.8087604217072752 0loss/FBt2/assignment_loss1.6378330318534258 _runtime2200.601474299 _step101 0loss/FRt1/assignment_loss1.4285997706790303 1loss/SRqqt2/detection_loss2.0437043278023252 $ _timestamp1.788467310561101e+09 1loss/SRqqt2/assignment_loss2.048653479925422 /loss/FRt2/assignment_loss3.668356844748574 /loss/FRt3/assignment_loss3.115030294491185 1loss/FBt1/assignment_loss0.09223904733817824 0loss/FBt4/assignment_loss0.0945895829286706 .loss/FBt2/detection_loss2.744627989255465 0loss/FBt3/assignment_loss0.2919867044858014 /loss/FRt3/detection_loss2.6110864323046474e (: ʳ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts253.262' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent50.6524 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors08?۰p (:  disk.sda3.out 66.78515625 disk.dm-0.in 5128.48828125 disk.sda1.in0 disk.sda2.out0 disk.dm-1.out 66.78515625 disk.drbd1260.out 0.3046875% disk./.usageGB1151.1238441467285" disk.dm-0.out50659.2666015625' disk.nvme0c0n1.out50659.2666015625 disk.drbd1260.in0$ memory_percent77.87275314331055' proc.memory.availableMB 3625.328125 proc.memory.rssMB3124.875 cpu25.02044148767426 disk.sda.in0 disk.sda.out 66.78515625 network.sent 2.650268e+06 network.recv716044) disk./.usagePercent77.25288703792484% disk.nvme0n1.out50659.2666015625 disk.sda1.out0 disk.sda3.in0! disk.nvme0n1.in 5128.48828125 disk.sda2.in0# disk.nvme0c0n1.in 5128.48828125 disk.dm-1.in0* proc.memory.percent19.072723388671875 proc.cpu.threads17J}t (: ф disk.drbd1260.in0# disk.dm-0.out50813.47314453125 disk.sda3.out 67.12890625 disk.sda.in0 disk.sda2.in0' proc.memory.availableMB 3544.671875 proc.memory.rssMB3124.875 network.recv722296) disk./.usagePercent77.25288857397004 disk.dm-1.out 67.12890625 disk.sda.out 67.12890625 network.sent 2.737604e+06 disk.dm-1.in0 disk.drbd1260.out 0.30859375& disk.nvme0n1.out50813.47314453125 disk.sda1.out0 disk.sda3.in0 disk.sda2.out0$ memory_percent78.36503982543945# disk.nvme0c0n1.in 5128.48828125( disk.nvme0c0n1.out50813.47314453125! disk.nvme0n1.in 5128.48828125 disk.dm-0.in 5128.48828125 disk.sda1.in0* proc.memory.percent19.072723388671875 cpu25.022904060284134 proc.cpu.threads17$ disk./.usageGB1151.1238670349122! (: ˟ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts240.411' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.0822 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 (:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp50.0 gpu.0.powerWatts336.22' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.244 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0H@s (: " disk./.usageGB1150.6962890625& disk.nvme0n1.out51065.63720703125 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0 disk.dm-1.out 67.57421875) proc.memory.availableMB 3448.16796875 network.sent 2.74334e+06 disk.dm-0.in 5128.48828125 disk.sda3.out 67.57421875 disk.sda.out 67.57421875# disk.nvme0c0n1.in 5128.48828125 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.30859375 disk.sda.in0! disk.nvme0n1.in 5128.48828125# disk.dm-0.out51065.63720703125 disk.sda3.in0 disk.sda2.out0( disk.nvme0c0n1.out51065.63720703125 proc.memory.rssMB3124.875) disk./.usagePercent77.22419345747966$ memory_percent78.95405292510986* proc.memory.percent19.072723388671875 cpu25.025566313896427 proc.cpu.threads17 network.recv726721( (:  gpu.0.gpu97.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts122.504' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent24.5008 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0El (:  disk.sda3.out 67.8984375 disk.sda2.out0 disk.dm-1.out 67.8984375) proc.memory.availableMB 3347.80859375 proc.memory.rssMB3124.875 cpu25.027943075789196 network.sent 2.749125e+06 disk.sda1.in0 disk.sda.out 67.8984375 disk.sda2.in0" disk.nvme0c0n1.in 5276.1171875 disk.drbd1260.in0* proc.memory.percent19.072723388671875 proc.cpu.threads17) disk./.usagePercent77.17494119199729 disk.nvme0n1.in 5276.1171875 disk.sda3.in0 disk.dm-1.in0 network.recv731000% disk./.usageGB1149.9623947143555& disk.nvme0n1.out51340.08154296875 disk.sda1.out0 disk.sda.in0( disk.nvme0c0n1.out51340.08154296875 disk.drbd1260.out0.3125# memory_percent79.5665979385376 disk.dm-0.in 5276.1171875# disk.dm-0.out51340.08154296875Ɠ (:  gpu.0.gpu97.0 gpu.0.memory21+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp44.0 gpu.0.powerWatts139.406' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent27.8812 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0}'g (:  proc.cpu.threads17 network.sent 2.754039e+06) disk./.usagePercent77.17980226305745 disk.nvme0n1.in 5276.1171875& disk.nvme0n1.out52015.70654296875 disk.dm-0.in 5276.1171875 disk.sda1.in0 disk.sda3.out68.40625 disk.sda.out68.40625 network.recv734414% disk./.usageGB1150.0348281860352 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.dm-1.out68.40625 disk.drbd1260.in0 disk.drbd1260.out0.3125 disk.sda1.out0 disk.sda3.in0 disk.sda.in0" disk.nvme0c0n1.in 5276.1171875( disk.nvme0c0n1.out52015.70654296875$ memory_percent80.17847537994385) proc.memory.availableMB 3247.55859375* proc.memory.percent19.072723388671875# disk.dm-0.out52015.70654296875 proc.memory.rssMB3124.875 cpu25.030250278343953库 (:  gpu.0.gpu93.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts241.258' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.2516 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0:q (:  disk.sda.out 68.75390625 disk.sda2.in0 disk.dm-1.in0 disk.nvme0n1.in 5276.1171875* proc.memory.percent19.072723388671875 cpu25.032241401207816 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.sda2.out0" disk.nvme0c0n1.in 5276.1171875% disk.nvme0n1.out52148.3818359375 disk.dm-0.in 5276.1171875$ memory_percent80.81467151641846 network.recv738870) disk./.usagePercent77.18138157353619% disk./.usageGB1150.0583610534668 disk.sda3.out 68.75390625 disk.drbd1260.in0 disk.drbd1260.out 0.31640625" disk.dm-0.out52148.3818359375) proc.memory.availableMB 3143.32421875 network.sent 2.761944e+06 disk.sda3.in0' disk.nvme0c0n1.out52148.3818359375 disk.dm-1.out 68.75390625 proc.memory.rssMB3124.875 proc.cpu.threads17BK2 - _wandbruntime2282  _runtime2282\2  % _timestamp1.7884673922505217e+09 _runtime2282.291174088 _step102 & lr-AdamW/pg10.0011308876767887285 & lr-AdamW/pg20.0011308876767887285 trainer/global_step1299    & lr-AdamW/pg20.0011308876767887285 trainer/global_step1299 % _timestamp1.7884673922505217e+09 _runtime2282.291174088 _step102 & lr-AdamW/pg10.0011308876767887285fQ2 - _wandbruntime2282  _runtime2282Sǻ O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated61609937317376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated274757599935519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3017747227648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed286283456197120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated283268250760192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated18710848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5765513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated24447767 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 \HDe@n !viceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed283268225332224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current477 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated286285997987840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5765510 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated43158615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated286285997987840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 % _timestamp1.7884673922519915e+09 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed18710842 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated15003832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed274755060139143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed24428084 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9237842 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3013783417775 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated57751311207936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3015230864896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3011269049367 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed43138926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated283268250760192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3015230864896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3017747227648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed18710842 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated24447767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed57751282109952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated271743816517744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed286283456197120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed61609847182336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed271743791089776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed24428084 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated18710848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9238319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 trainer/global_step1299 _step103 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 _runtime2282.292816725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed15003352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed283268225332224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0' O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 _runtime2282.292816725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed15003352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed283268225332224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated61609937317376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated274757599935519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3017747227648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed286283456197120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated283268250760192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated18710848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5765513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated24447767 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed283268225332224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current477 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated286285997987840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed5765510 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated43158615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated286285997987840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 % _timestamp1.7884673922519915e+09 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed18710842 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated15003832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed274755060139143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed24428084 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9237842 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90135040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3013783417775 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated57751311207936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3015230864896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3011269049367 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed43138926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated283268250760192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3015230864896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3017747227648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed18710842 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated24447767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed57751282109952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated271743816517744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed286283456197120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed61609847182336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed271743791089776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed24428084 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated18710848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current61037056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9238319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 trainer/global_step1299 _step103g-2 - _wandbruntime2283  _runtime2283"*r+& M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime2283.813559744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated283479241135616 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed15014545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3013541304221 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated271946345227328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 trainer/global_step1299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated274962400900165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed271946319799360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated57795372577280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated43190873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated286499263592960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed283479215707648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3020022457344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated43190873 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated61656900521472 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 trainer/global_step1299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated274962400900165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed271946319799360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated57795372577280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated43190873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated286499263592960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed283479215707648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3020022457344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated43190873 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated61656900521472 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime2283.813559744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated283479241135616 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed15014545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3013541304221 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated271946345227328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3129019512320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated25368669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9585772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated44773567 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current459 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed25348986 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated25368669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated293817769531392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4004206966784 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3124908964075 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed59954350577152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated284997675329427 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed296946763615744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed296946763615744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed44753878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated281870251996944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3131535875072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5982113 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9585313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated293817769531392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated296949305406464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _runtime2365.809985287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed15567423 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884674757673934e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated296949305406464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed284995135533051 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated19404898 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3129019512320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated15567885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed63958498604032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated59954379675136 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed281870226568976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3127423332483 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed25348986 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 trainer/global_step1349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104jK O H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 trainer/global_step1349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3129019512320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated25368669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9585772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated44773567 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current459 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed25348986 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated25368669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated293817769531392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4004206966784 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3124908964075 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed59954350577152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated284997675329427 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed296946763615744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed296946763615744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 E8Dev iceStatsMonitor.on_train_batch_start/active.all.freed44753878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated281870251996944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3131535875072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated5982113 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9585313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated293817769531392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated296949305406464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 _runtime2365.809985287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed15567423 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884674757673934e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated296949305406464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed284995135533051 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated19404898 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3129019512320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated15567885 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed63958498604032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated59954379675136 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed281870226568976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3127423332483 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed25348986k=į"2 - _wandbruntime2367  _runtime2367ҥ& M RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated282072780706528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated59998441044480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed285199936497489 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3129695587545 trainer/global_step1349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated297162571011584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3131294728704 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated44805825 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated19418779 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3131294728704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed19418773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed19418773 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3133811104768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed15578577 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed294028734478848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed59998411946496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated64005565051392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4007065080320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated9592591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 _step108 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated19418779 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed25367337 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated15579036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4007124006912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated25387046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated294028759906816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated285202476294073 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 369380114618J& M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated25387046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated294028759906816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated285202476294073 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated282072780706528 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated59998441044480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed285199936497489 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3129695587545 trainer/global_step1349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated297162571011584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3131294728704 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated44805825 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated19418779 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3131294728704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed19418773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed19418773 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3133811104768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed15578577 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed294028734478848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed59998411946496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated64005565051392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4007065080320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated9592591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 _step108 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated19418779 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed25367337 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated15579036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4007124006912lŏ2 - _wandbruntime2367  _runtime2367&uX  2loss/SRqqt3/assignment_loss1.8269372385387357 2loss/SRqqt4/assignment_loss1.7484160304142564 1loss/SRqqt3/detection_loss2.3252005475132087 trainer/global_step1349 $ _timestamp1.788467477328722e+09 1loss/FBt2/assignment_loss0.06622113606438991 /loss/FRt1/detection_loss1.6268655361547586 1loss/SRqqt4/detection_loss1.9344438029562725 .loss/FBt2/detection_loss2.824365375439326 .loss/FBt3/detection_loss2.518889047205448 _step109 /loss/FRt3/assignment_loss4.119842591456004 .loss/FRt3/detection_loss2.753587348972048 /loss/FBt4/detection_loss2.2554859121640525 /loss/FRt1/assignment_loss1.354640576147264 0loss/FBt1/assignment_loss0.1352328676619242 .loss/FRt4/detection_loss2.394351716400409 1loss/SRqqt1/detection_loss1.0081256108431074 1loss/SRqqt2/detection_loss1.8637165845534123 %loss/total_loss44.08601148817184 epoch2 2loss/SRqqt2/assignment_loss1.7088322474176936 .loss/FRt2/detection_loss2.243322482953469 _runtime2367.369306962 0loss/FRt2/assignment_loss3.3063238790879645 /loss/FRt4/assignment_loss3.655750105575639 1loss/SRqqt1/assignment_loss1.128834781890082 0loss/FBt4/assignment_loss0.5600161786611092 0loss/FBt1/detection_loss0.47685565779773664 1loss/FBt3/assignment_loss0.249754232303530447S\  1loss/FBt3/assignment_loss0.24975423230353044 2loss/SRqqt3/assignment_loss1.8269372385387357 2loss/SRqqt4/assignment_loss1.7484160304142564 1loss/SRqqt3/detection_loss2.3252005475132087 trainer/global_step1349 $ _timestamp1.788467477328722e+09 1loss/FBt2/assignment_loss0.06622113606438991 /loss/FRt1/detection_loss1.6268655361547586 1loss/SRqqt4/detection_loss1.9344438029562725 .loss/FBt2/detection_loss2.824365375439326 .loss/FBt3/detection_loss2.518889047205448 _step109 /loss/FRt3/assignment_loss4.119842591456004 .loss/FRt3/detection_loss2.753587348972048 /loss/FBt4/detection_loss2.2554859121640525 /loss/FRt1/assignment_loss1.354640576147264 0loss/FBt1/assignment_loss0.1352328676619242 .loss/FRt4/detection_loss2.394351716400409 1loss/SRqqt1/detection_loss1.0081256108431074 1loss/SRqqt2/detection_loss1.8637165845534123 %loss/total_loss44.08601148817184 epoch2 2loss/SRqqt2/assignment_loss1.7088322474176936 .loss/FRt2/detection_loss2.243322482953469 _runtime2367.369306962 0loss/FRt2/assignment_loss3.3063238790879645 /loss/FRt4/assignment_loss3.655750105575639 1loss/SRqqt1/assignment_loss1.128834781890082 0loss/FBt4/assignment_loss0.5600161786611092 0loss/FBt1/detection_loss0.47685565779773664m~ (:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts235.38' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.076 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0dQr (: ᘥ disk.sda.out 71.61328125# disk.nvme0c0n1.in 5494.87890625( proc.memory.availableMB 2522.8359375 proc.cpu.threads17 network.recv763891% disk./.usageGB1149.8863105773926! disk.nvme0n1.in 5494.87890625 disk.sda1.in0 disk.sda3.in0( disk.nvme0c0n1.out53963.65966796875 disk.drbd1260.out0.328125$ memory_percent84.60183143615723) disk./.usagePercent77.16983512172688 disk.dm-0.in 5494.87890625# disk.dm-0.out53963.65966796875 disk.sda1.out0 disk.sda3.out 71.61328125 disk.drbd1260.in0* proc.memory.percent19.072723388671875 cpu25.04614741200014 network.sent 2.88719e+06& disk.nvme0n1.out53963.65966796875 disk.sda.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.dm-1.out 71.61328125 proc.memory.rssMB3124.875l8 (: Զ gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts242.322' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent48.464400000000005 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0qj (:  disk.nvme0n1.in 5614.109375 disk.dm-0.in 5614.109375# disk.dm-0.out54041.34228515625 disk.sda2.out0 network.recv768886 disk.drbd1260.out0.328125& disk.nvme0n1.out54041.34228515625 disk.sda1.in0 disk.sda3.out 71.9609375* proc.memory.percent19.072723388671875( disk.nvme0c0n1.out54041.34228515625 disk.dm-1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda.out 71.9609375$ memory_percent85.16459465026855( proc.memory.availableMB 2430.6328125 proc.memory.rssMB3124.875% disk./.usageGB1149.8911094665527 disk.dm-1.out 71.9609375 disk.drbd1260.in0 disk.sda.in0 disk.sda2.in0 cpu25.048477179415084 proc.cpu.threads17 network.sent 2.96229e+06) disk./.usagePercent77.17015717920495! disk.nvme0c0n1.in 5614.109375v@ (:  gpu.0.gpu93.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts242.604' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.5208 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0l (:  disk.sda2.out0' disk.nvme0c0n1.out54354.7587890625 network.sent 2.966407e+06 disk.drbd1260.in0! disk.nvme0c0n1.in 5614.109375# memory_percent85.7792615890503) proc.memory.availableMB 2329.92578125 cpu25.05077037208534 proc.cpu.threads17 network.recv773539 disk.dm-1.in0 disk.nvme0n1.in 5614.109375% disk.nvme0n1.out54354.7587890625" disk.dm-0.out54354.7666015625 proc.memory.rssMB3124.875) disk./.usagePercent77.17015717920495% disk./.usageGB1149.8911094665527 disk.dm-1.out 72.33203125 disk.drbd1260.out 0.33203125 disk.sda.in0 disk.sda.out 72.33203125* proc.memory.percent19.072723388671875 disk.dm-0.in 5614.109375 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 72.33203125 disk.sda2.in0Gq*Ē (: IJ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts230.835' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.167 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors02] (: IJ disk.dm-1.out 72.703125 disk.dm-0.out 54677.1015625 disk.sda1.in0 proc.memory.rssMB3124.875% disk./.usageGB1150.0162963867188 disk.sda2.in0$ disk.nvme0c0n1.out 54677.1015625 disk.dm-1.in0 disk.nvme0n1.in 5614.109375" disk.nvme0n1.out 54677.1015625 proc.cpu.threads17) disk./.usagePercent77.17855857845612 disk.drbd1260.in0 disk.dm-0.in 5614.109375 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 72.703125$ memory_percent86.38811111450195' proc.memory.availableMB 2230.171875 network.recv777549 disk.sda.in0 disk.sda.out 72.703125 disk.sda2.out0 disk.drbd1260.out 0.33203125* proc.memory.percent19.072723388671875 cpu25.053020596143572 network.sent 2.973913e+06! disk.nvme0c0n1.in 5614.109375yk (: Ӳط disk.sda3.in0 disk.sda.out 73.19140625 cpu25.055294829148973 proc.cpu.threads17& disk.nvme0n1.out55096.82958984375 disk.dm-0.in 5614.109375 disk.sda3.out 73.19140625 disk.sda2.out0( disk.nvme0c0n1.out55096.82958984375 disk.dm-1.out 73.19140625$ memory_percent87.02583312988281 network.sent 2.979571e+06 network.recv781804( disk./.usagePercent77.1853870674169 disk.nvme0n1.in 5614.109375 disk.sda.in0 disk.sda2.in0! disk.nvme0c0n1.in 5614.109375 disk.dm-1.in0% disk./.usageGB1150.1180458068848# disk.dm-0.out55096.82958984375 disk.sda1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.3359375% proc.memory.availableMB 2125.6875 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875 disk.sda1.out0i (: Ӳ gpu.0.gpu80.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts325.144' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.0288 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrorVR*s0# gpu.0.uncorrectedMemoryErrors0/ (:  gpu.0.gpu72.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts248.469' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.693799999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0)rv (: ɲ disk.drbd1260.in0 disk.sda1.in0 disk.sda2.out0# disk.nvme0c0n1.in 5732.16796875 disk.dm-1.in0 disk.dm-1.out 73.57421875$ memory_percent87.58738040924072 cpu25.057328596838374 network.recv785464! disk.nvme0n1.in 5732.16796875& disk.nvme0n1.out55477.52099609375 disk.sda1.out0 disk.sda.in0 proc.cpu.threads17 network.sent 2.984587e+06) disk./.usagePercent77.18695946569221% disk./.usageGB1150.1414756774902 disk.dm-0.in 5732.16796875# disk.dm-0.out55477.52099609375 disk.sda3.out 73.57421875 disk.sda.out 73.57421875 disk.sda2.in0 disk.drbd1260.out 0.3359375 disk.sda3.in0( disk.nvme0c0n1.out55477.52099609375) proc.memory.availableMB 2033.68359375 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875-D&2 - _wandbruntime2449  _runtime2449ٝ  & lr-AdamW/pg20.0011166853560801084 trainer/global_step1399 % _timestamp1.7884675597676585e+09 _runtime2449.809011411 _step110 & lr-AdamW/pg10.0011166853560801084";  & lr-AdamW/pg10.0011166853560801084 & lr-AdamW/pg20.0011166853560801084 trainer/global_step1399 % _timestamp1.7884675597676585e+09 _runtime2449.809011411 _step110nl 2 - _wandbruntime2449  _runtime2449'B' O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6198713 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed46368830 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated295237750723335 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed307610071034368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed62157419044352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 trainer/global_step1399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed20098942 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated304367288302592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3245324522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated307612612825088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3245324522496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated307612612825088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated304367288302592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9933142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated46388519 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9932678 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed16131388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4149689932800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 $ _timestamp1.788467559768421e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated26289571 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed20098942 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated62157448142336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6198710 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4149633090048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3238548878783 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3242808159744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated26289571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed295235210926959 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated291996687476144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed26269888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3242808159744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed26269888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed304367262874624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated20098948 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed291996662048176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _runtime 2449.80969512 _step111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated20098948 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed66307052134400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3245324522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated307612612825088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3245324522496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated307612612825088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated304367288302592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated9933142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated46388519 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed9932678 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed16131388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4149689932800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 $ _timestamp1.788467559768421e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated26289571 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed20098942 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated62157448142336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6198710 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4149633090048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3238548878783 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3242808159744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated26289571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed295235210926959 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated291996687476144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed26269888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3242808159744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed26269888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed304367262874624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated20098948 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed291996662048176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _runtime 2449.80969512 _step111 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated20098948 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed66307052134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed307610071034368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed62157419044352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 trainer/global_step1399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed20098942 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated304367288302592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888ovA~2 - _wandbruntime2451  _runtime2451<(& M [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated304578278678016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3245083376128 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed20112823 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed66354028733440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current471 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated66354116758016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed6203042 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated292199216185728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated46420777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 % _timestamp1.7884675612660496e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated20112829 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed26288239 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 3247599752192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current468 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3245083376128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3247599752192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated20112829 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3240821133637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed304578253250048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed26288239 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated62201509511680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated16143122 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated307825878430208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated304578278678016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed16142651 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated295442551687981 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed9939609 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4152548319744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated26307948 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated9940077 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated26307948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4152607246336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed292199190757760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 trainer/global_step1399 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _runtime2451.307101519 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed304578253250048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated46420777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 % _timestamp1.7884675612660496e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated20112829 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed26288239 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 3247599752192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current468 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3245083376128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3247599752192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated20112829 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3240821133637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed304578253250048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed26288239 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated62201509511680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated16143122 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated307825878430208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated304578278678016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed16142651 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated295442551687981 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed9939609 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4152548319744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated26307948 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated9940077 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated26307948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4152607246336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed292199190757760 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 trainer/global_step1399 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _runtime2451.307101519 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed304578253250048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated318275920243712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3359113169920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated16695636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed27190790 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed16695177 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed47983782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed20792992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3359113169920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4295214874624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed314916781645824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3354703161899 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated20792998 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 _step115 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated305477826117243 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated302123122955344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3352188793491 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed318273378452992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated314916807073792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed314916781645824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed20792992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10280323 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6415313 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 % _timestamp1.7884676438762608e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed68655702386176gD7  O D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated305477826117243 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated302123122955344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3352188793491 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed318273378452992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated314916807073792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed314916781645824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed20792992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10280323 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6415313 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 % _timestamp1.7884676438762608e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed68655702386176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 trainer/global_step1449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bgjytes.all.current 2541790720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6415310 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3356596807168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated314916807073792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3356596807168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated48003471 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed10279867 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated318275920243712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3359113169920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated16695636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed27190790 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed16695177 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed47983782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed20792992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3359113169920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4295214874624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed314916781645824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3354703161899 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated20792998 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 _step115sdo (:  disk.dm-0.in 5871.1953125 disk.sda1.out0 disk.sda.out 76.14453125 disk.sda2.in0$ memory_percent80.32195568084717) proc.memory.availableMB 3224.05078125* proc.memory.percent19.072723388671875 network.recv812073$ disk./.usageGB1150.891529083252 disk.sda3.in0 disk.sda.in0' disk.nvme0c0n1.out57475.9853515625 disk.dm-1.out 76.14453125 disk.drbd1260.out 0.34765625 proc.cpu.threads17 disk.nvme0n1.in 5871.1953125% disk.nvme0n1.out57475.9853515625" disk.dm-0.out57475.9853515625 disk.sda1.in0 disk.sda3.out 76.14453125 disk.drbd1260.in0 network.sent 3.097876e+06( disk./.usagePercent77.2372961790896 disk.sda2.out0" disk.nvme0c0n1.in 5871.1953125 disk.dm-1.in0 proc.memory.rssMB3124.875 cpu25.071745608594117T9R (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts237.776' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.555200000000006 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Y%*2 - _wandbruntime2535  _runtime25357_p& M RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed48016014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated305682627081889 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated27228850 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed20806873 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3361388399616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed27209141 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated302325651664928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3356975416961 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3354461048345 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated318489185848832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed318486644044800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated64404577978880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated20806879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed6419642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated48035729 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed302325626236960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed16706326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4298136124416 _runtime2535.472684955 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 3361388399616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed68702685005312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated315127797449216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed48016014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3358872023552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed27209141 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated68702770932736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed315127772021248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated315127797449216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated48035729 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated16706781 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed20806873 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829440 trainer/global_step1449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed315127772021248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated6419645 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated27228850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3}O' M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated27228850 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed20806873 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3361388399616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed27209141 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated302325651664928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3356975416961 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3354461048345 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated318489185848832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed318486644044800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated64404577978880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated20806879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed6419642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated48035729 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed302325626236960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed16706326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4298136124416 _runtime2535.472684955 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 3361388399616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed68702685005312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated315127797449216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed48016014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3358872023552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed27209141 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated68702770932736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed315127772021248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated315127797449216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated48035729 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated16706781 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed20806873 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829440 trainer/global_step1449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed315127772021248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated6419645 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated27228850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed48016014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated305682627081889t2 - _wandbruntime2535  _runtime2535 0X  1loss/FBt1/assignment_loss0.17195101242030866 0loss/FBt4/assignment_loss0.3168476765806025 /loss/FRt4/detection_loss2.1528303745118054 0loss/FRt1/assignment_loss1.9296785897264879 2loss/SRqqt4/assignment_loss1.6927233068272471 2loss/SRqqt2/assignment_loss1.4763769859329183 .loss/FRt2/detection_loss2.551145409476267 -loss/FRt3/detection_loss2.91892172396183 trainer/global_step1449 2loss/SRqqt1/assignment_loss1.0895969575939288 /loss/FBt1/detection_loss0.7470981217535201 .loss/FBt3/detection_loss3.059409187390254 epoch2 /loss/FRt2/assignment_loss3.679749897081558 1loss/SRqqt1/detection_loss1.2077402519986375 %loss/total_loss46.21026129985058 0loss/FRt3/assignment_loss3.6049082607030867 1loss/FBt3/assignment_loss0.28379097709862083 1loss/SRqqt4/detection_loss1.8451978737399692 .loss/FBt2/detection_loss2.609737458825111 /loss/FBt4/detection_loss3.4339438405903904 .loss/FRt1/detection_loss1.832092869405945 _step117 2loss/SRqqt3/assignment_loss1.6705331649382908 1loss/FBt2/assignment_loss0.22660563598619773 1loss/SRqqt2/detection_loss2.1721074283123016 1loss/SRqqt3/detection_loss2.3471587336063386 $ _timestamp1.788467645433933e+09 _runtime2535.474295953 .loss/FRt4/assignment_loss3.19011556138896A…\  .loss/FRt4/assignment_loss3.19011556138896 1loss/FBt1/assignment_loss0.17195101242030866 0loss/FBt4/assignment_loss0.3168476765806025 /loss/FRt4/detection_loss2.1528303745118054 0loss/FRt1/assignment_loss1.9296785897264879 2loss/SRqqt4/assignment_loss1.6927233068272471 2loss/SRqqt2/assignment_loss1.4763769859329183 .loss/FRt2/detection_loss2.551145409476267 -loss/FRt3/detection_loss2.91892172396183 trainer/global_step1449 2loss/SRqqt1/assignment_loss1.0895969575939288 /loss/FBt1/detection_loss0.7470981217535201 .loss/FBt3/detection_loss3.059409187390254 epoch2 /loss/FRt2/assignment_loss3.679749897081558 1loss/SRqqt1/detection_loss1.2077402519986375 %loss/total_loss46.21026129985058 0loss/FRt3/assignment_loss3.6049082607030867 1loss/FBt3/assignment_loss0.28379097709862083 1loss/SRqqt4/detection_loss1.8451978737399692 .loss/FBt2/detection_loss2.609737458825111 /loss/FBt4/detection_loss3.4339438405903904 .loss/FRt1/detection_loss1.832092869405945 _step117 2loss/SRqqt3/assignment_loss1.6705331649382908 1loss/FBt2/assignment_loss0.22660563598619773 1loss/SRqqt2/detection_loss2.1721074283123016 1loss/SRqqt3/detection_loss2.3471587336063386 $ _timestamp1.788467645433933e+09 _runtime2535.474295953uk8m (: ˳϶ disk.drbd1260.in0 disk.nvme0n1.in 5871.1953125% disk.nvme0n1.out57727.4111328125" disk.dm-0.out57727.4111328125 disk.sda1.out0 disk.sda.in0 disk.sda2.out0' disk.nvme0c0n1.out57727.4111328125 network.sent 3.17367e+06% disk./.usageGB1150.8914527893066 disk.dm-0.in 5871.1953125 disk.sda2.in0$ memory_percent80.96117973327637* proc.memory.percent19.072723388671875 cpu25.07403880628449 proc.cpu.threads17 network.recv818081) disk./.usagePercent77.23729105893892 disk.dm-1.in0 disk.drbd1260.out 0.3515625 disk.sda3.out 76.74609375 proc.memory.rssMB3124.875 disk.dm-1.out 76.74609375 disk.sda1.in0 disk.sda3.in0 disk.sda.out 76.74609375" disk.nvme0c0n1.in 5871.1953125( proc.memory.availableMB 3119.3203125 (: ˳ gpu.0.gpu95.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts228.679' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.7358 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0m{r (: ڳ disk.nvme0n1.in 5984.0390625 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda.out 77.265625" disk.dm-0.out57905.3564453125) proc.memory.avail<ableMB 3019.08203125 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875 network.sent 3.249342e+06 disk.sda2.out0" disk.nvme0c0n1.in 5984.0390625' disk.nvme0c0n1.out57905.3564453125 disk.dm-1.out 77.265625$ memory_percent81.57298564910889 cpu25.076470008873628% disk./.usageGB1150.8914527893066 disk.sda1.in0% disk.nvme0n1.out57905.3564453125 disk.dm-0.in 5984.0390625 proc.cpu.threads17 network.recv823140) disk./.usagePercent77.23729105893892 disk.sda3.out 77.265625 disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.3515625{: (: ڳ gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts237.516' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.5032 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Rxq (: 芯" disk.nvme0c0n1.in 5984.0390625( disk.nvme0c0n1.out58479.74853515625 disk.dm-1.out 77.69921875) disk./.usagePercent77.23571610058826 disk.nvme0n1.in 5984.0390625 disk.sda3.out 77.69921875 disk.sda.in0 disk.sda.out 77.69921875 disk.drbd1260.in0$ memory_percent82.21073150634766* proc.memory.percent19.072723388671875 network.recv827746% disk./.usageGB1150.8679847717285 disk.dm-0.in 5984.0390625 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0& proc.memory.availableMB 2914.59375# disk.dm-0.out58479.74853515625 disk.drbd1260.out 0.35546875 proc.memory.rssMB3124.875 cpu25.078997066062527 proc.cpu.threads17 network.sent 3.255315e+06& disk.nvme0n1.out58479.74853515625 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0ǁ (:  gpu.0.gpu62.0 gpu.0.memory14+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts326.2' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.24 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0[Lp (: ( disk.nvme0c0n1.out58754.87744140625% disk./.usageGB1150.8914527893066 disk.sda3.in0 disk.sda3.out 78.1328125 disk.sda.in0 disk.drbd1260.in0 disk.drbd1260.out 0.35546875$ memory_percent82.81958103179932) proc.memory.availableMB 2814.83984375& disk.nvme0n1.out58754.87744140625 disk.sda.out 78.1328125 disk.sda2.out0" disk.nvme0c0n1.in 5984.0390625 disk.dm-1.in0 disk.dm-1.out 78.1328125* proc.memory.percent19.072723388671875 proc.cpu.threads17) disk./.usagePercent77.23729105893892 disk.sda2.in0 proc.memory.rssMB3124.875 cpu25.081164853968605 network.sent 3.262811e+06 network.recv831926 disk.nvme0n1.in 5984.0390625 disk.dm-0.in 5984.0390625# disk.dm-0.out58754.87744140625 disk.sda1.in0 disk.sda1.out0\+] (:  gpu.0.gpu89.0 gpu.0.memory21+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts194.47' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent38.894 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0c (: ѝ% disk./.usageGB1150.8914527893066 disk.sda3.in0 disk.sda.in0% disk.nvme0c0n1.out59198.33203125* proc.memory.percent19.072723388671875 cpu25.083580175692298 disk.nvme0n1.in 5984.0390625# disk.nvme0n1.out59198.33203125 disk.dm-0.out59198.33203125 disk.dm-1.in0 network.sent 3.266586e+06 disk.sda1.in0 disk.sda3.out 78.8984375" disk.nvme0c0n1.in 5984.0390625 disk.dm-1.out 78.8984375 disk.drbd1260.in0' proc.memory.availableMB 2710.421875 proc.memory.rssMB3124.875 disk.dm-0.in 5984.0390625 disk.sda1.out0 disk.sda.out 78.8984375 disk.sda2.in0 disk.sda2.out0 disk.drbd1260.out0.359375# memory_percent83.4568977355957 proc.cpu.threads17 network.recv835740) disk./.usagePercent77.23729105893892)nYL (:  gpu.0.gpu27.0 gpu.0.memory8+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts331.561' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.3122 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors02 - _wandbruntime2618  _runtime2618n  & lr-AdamW/pg10.0011007206684223172 & lr-AdamW/pg20.0011007206684223172 trainer/global_step1499 % _timestamp1.7884677281439323e+09 _runtime2618.184625519 _step118JC;N  _step118 & lr-AdamW/pg10.0011007206684223172 & lr-AdamW/pg20.0011007206684223172 trainer/global_step1499 % _timestamp1.7884677281439323e+09 _runtime2618.184625519v'2 - _wandbruntime2618  _runtime2618I' O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated66563585076736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3472901817344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated21487048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated325466325844992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10628012 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated21487048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated49618423 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4440947344384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed21487042 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed28111692 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3470385454592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed21487042 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed66563555978752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3465828708199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 _runtime2618.186606304 _step119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed325466300417024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated17259925 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed312249533006576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 trainer/global_step1499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated28131375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed10627539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated71004532421120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 $ _timestamp1.788467728145417e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated328939227662336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed328936685871616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6631913 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed49598734 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3468343076607 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed315715361714775 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated328939227662336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated325466325844992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3470385454592 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed71004444383232 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated28131375 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed328936685871616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3472901817344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6631910 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4440888404480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated312249558434544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248#' O UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed315715361714775 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated328939227662336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated325466325844992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3470385454592 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed71004444383232 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated28131375 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed328936685871616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3472901817344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6631910 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4440888404480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated312249558434544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated66563585076736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3472901817344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated21487048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated325466325844992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10628012 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated21487048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated49618423 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4440947344384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed21487042 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed28111692 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3470385454592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed21487042 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed66563555978752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3465828708199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 _runtime2618.186606304 _step119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed325466300417024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated17259925 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed312249533006576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 trainer/global_step1499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated28131375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed10627539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated71004532421120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 $ _timestamp1.788467728145417e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated328939227662336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed328936685871616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6631913 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed49598734 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3468343076607 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104w2 - _wandbruntime2619  _runtime2619PfZ M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4443842768896 trainer/global_step1499 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed66607617348096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated329152493267456 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 _runtime2619.665203874 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed315920162679213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3472660670976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed312452061716160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated66607646446080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 _step120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed17270669 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed21500923 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current479 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3470615331669 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3468100963053 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed325677290792448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated28149752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated17271148 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated312452087144128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3475177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current476 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed28130043 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed49630966 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated71051489214976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UIDeviceStatsMonitor.on_train_batchpޜ _end/inactive_split.large_pool.allocated6636245 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed28130043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed49630966 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated49650681 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated49650681 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed329149951463424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 % _timestamp1.7884677296240087e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed10634427 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4443783842304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated315922702475797 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed325677290792448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3472660670976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed312452061716160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated66607646446080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 _step120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed17270669 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed21500923 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current479 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3470615331669 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3468100963053 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed325677290792448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated28149752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated17271148 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated312452087144128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3475177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current476 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed28130043 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed49630966 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated71051489214976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated6636245 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed28130043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed49630966 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated49650681 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated49650681 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808xu2 - _wandbruntime2619  _runtime2619#yV  /loss/FRt3/detection_loss2.9902074902744618 /loss/FRt4/detection_loss2.2028401942440756 1loss/SRqqt4/detection_loss1.8887901166211003 /loss/FBt2/detection_loss2.7860717905892267 0loss/FRt1/assignment_loss1.4873135956850918 /loss/FRt2/assignment_loss3.343979057015442 /loss/FRt3/assignment_loss2.901561360237962 1loss/SRqqt2/detection_loss2.1669973151927646 .loss/FBt3/detection_loss3.017017900943756 epoch2 _runtime2619.666260457 _step121 2loss/SRqqt1/assignment_loss1.2505786126195793 /loss/FBt3/assignment_loss0.314015378166611 /loss/FRt1/detection_loss1.8528028229872386 0loss/SRqqt3/detection_loss2.309418350458145 %loss/total_loss44.69027963573737 .loss/FRt2/detection_loss2.396718308329582 $ _timestamp1.788467729625869e+09 2loss/SRqqt2/assignment_loss2.0144685826856983 2loss/SRqqt3/assignment_loss1.8298763230531698 1loss/SRqqt4/assignment_loss1.582293841858273 1loss/SRqqt1/detection_loss0.9687374698997333 0loss/FBt2/assignment_loss0.7000904443622049 .loss/FRt4/assignment_loss2.98155871088858 1loss/FBt1/assignment_loss0.14481102582902206 0loss/FBt4/assignment_loss0.3915364809698076 /loss/FBt1/detection_loss0.6619742106934274 /loss/FBt4/detection_loss2.5066202521324157 trainer/global_step14995dAZ  .loss/FRt4/assignment_loss2.98155871088858 1loss/FBt1/assignment_loss0.14481102582902206 0loss/FBt4/assignment_loss0.3915364809698076 /loss/FBt1/detection_loss0.6619742106934274 /loss/FBt4/detection_loss2.5066202521324157 trainer/global_step1499 /loss/FRt3/detection_loss2.9902074902744618 /loss/FRt4/detection_loss2.2028401942440756 1loss/SRqqt4/detection_loss1.8887901166211003 /loss/FBt2/detection_loss2.7860717905892267 0loss/FRt1/assignment_loss1.4873135956850918 /loss/FRt2/assignment_loss3.343979057015442 /loss/FRt3/assignment_loss2.901561360237962 1loss/SRqqt2/detection_loss2.1669973151927646 .loss/FBt3/detection_loss3.017017900943756 epoch2 _runtime2619.666260457 _step121 2loss/SRqqt1/assignment_loss1.2505786126195793 /loss/FBt3/assignment_loss0.314015378166611 /loss/FRt1/detection_loss1.8528028229872386 0loss/SRqqt3/detection_loss2.309418350458145 %loss/total_loss44.69027963573737 .loss/FRt2/detection_loss2.396718308329582 $ _timestamp1.788467729625869e+09 2loss/SRqqt2/assignment_loss2.0144685826856983 2loss/SRqqt3/assignment_loss1.8298763230531698 1loss/SRqqt4/assignment_loss1.582293841858273 1loss/SRqqt1/detection_loss0.9687374698997333 0loss/FBt2/assignment_loss0.7000904443622049y!!h (: ۧ% disk.nvme0c0n1.out59448.35546875 disk.dm-1.in0$ memory_percent84.14416313171387 disk.dm-0.in 6073.8671875 disk.sda2.out0 disk.dm-1.out 79.26171875* proc.memory.percent19.072723388671875 cpu25.085926218310625 network.sent 3.271606e+06) disk./.usagePercent77.23729105893892 disk.sda3.out 79.26171875 disk.sda.in0 disk.sda.out 79.26171875 disk.drbd1260.in0 proc.cpu.threads17 network.recv839224 disk.nvme0n1.in 6073.8671875 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0 disk.drbd1260.out0.359375( proc.memory.availableMB 2597.8203125 proc.memory.rssMB3124.875% disk./.usageGB1150.8914527893066# disk.nvme0n1.out59448.35546875 disk.dm-0.out59448.35546875 disk.sda3.in0" disk.nvme0c0n1.in 6073.8671875( (:  gpu.0.gpu94.0 gpu.0.memory23+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts329.698' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.9396 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ݱu (:  proc.memory.rssMB3124.875 cpu25.08806546868508 proc.cpu.threads17 network.sent 3.357909e+06 network.recv844611) disk./.usagePercent77.23729105893892 disk.sda.out 79.94921875 disk.sda2.out0# disk.nvme0c0n1.in 6133.33984375( disk.nvme0c0n1.out59758.37939453125! disk.nvme0n1.in 6133.33984375 disk.sda.in0 disk.dm-1.out 79.94921875& disk.nvme0n1.out59758.37939453125 disk.dm-0.in 6133.33984375# disk.dm-0.out59758.37939453125 disk.sda1.in0 disk.sda3.in0 disk.sda3.out 79.94921875 disk.drbd1260.in0% disk./.usageGB1150.8914527893066( proc.memory.availableMB 2497.8203125* proc.memory.percent19.072723388671875 disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.out 0.36328125 disk.sda1.out0$ memory_percent84.75451469421387R (:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts252.727' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent50.545399999999994 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0qy (:  gpu.0.gpu62.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts332.103' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.4206 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0YŸb (: # disk.nvme0c0n1.in 6133.33984375& disk.nvme0c0n1.out59961.666015625% disk./.usageGB1149.8026008605957! disk.nvme0n1.in 6133.33984375 disk.sda3.out80.375 disk.sda2.out0 disk.dm-1.in0 disk.dm-1.out80.375) proc.memory.availableMB 2385.80078125 proc.memory.rssMB3124.875) disk./.usagePercent77.16421729239549 disk.drbd1260.in0$ memory_percent85.43822765350342* proc.memory.percent19.072723388671875 proc.cpu.threads17 network.recv848614$ disk.nvme0n1.out59961.666015625! disk.dm-0.out59961.666015625 disk.sda1.out0 disk.sda2.in0 disk.drbd1260.out 0.36328125 cpu25.090223696503053 network.sent 3.363407e+06 disk.dm-0.in 6133.33984375 disk.sda1.in0 disk.sda3.in0 disk.sda.in0 disk.sda.out80.375f<DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed339599993290240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10975581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed10975115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed68766624445952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed22181092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 $ _timestamp1.788467811404106e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _step123 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 trainer/global_step1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed29032594 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated336015844616192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed73353013061632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 LFDeviceStat "sMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 _runtime2701.445294097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current469 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated73353101099520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3586690464768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated68766653543936 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3579468622907 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 370608701449Qw' O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3584174102016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated339602535080960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3586690464768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated68766653543936 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3579468622907 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed51213686 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated22181098 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated325957976905059 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed325955437108683 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated29052277 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated339602535080960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated51233375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated17824094 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated29052277 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated22181098 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed17823625 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4586388615680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3584174102016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed6848510 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated6848513 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated322375993913744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3581982991315 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed22181092 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed322375968485776 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3586690464768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4586447555584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed339599993290240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated10975581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed10975115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed68766624445952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed22181092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 $ _timestamp1.788467811404106e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _step123 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 trainer/global_step1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed29032594 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated336015844616192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed73353013061632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 _runtime2701.445294097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current469 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated73353101099520 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4589361454080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime2702.972080855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated336226834991616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3586449318400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated339815800686080 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current468 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated339815800686080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed22194973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3588965694464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3586449318400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated51265633 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed17834813 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed68810685815296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated10982439 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current471 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated29070654 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated322578522623328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated336226834991616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed6852842 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed336226809563648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated51265633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed73399988342784 trainer/global_step1549 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed326160238073121 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated22194979 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated17835284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed10981971 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed51245918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated29070654 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3584255246377 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 G' M EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed17834813 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed68810685815296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated10982439 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current471 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated29070654 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated322578522623328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated336226834991616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed6852842 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed336226809563648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated51265633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed73399988342784 trainer/global_step1549 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed326160238073121 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated22194979 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated17835284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed10981971 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed51245918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated29070654 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3584255246377 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4589361454080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime2702.972080855 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated336226834991616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3586449318400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated339815800686080 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current468 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated339815800686080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed22194973 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3588965694464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3586449318400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated51265633 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 _step127 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed22875142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed29953496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed29953496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7065113 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4731883591168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed346565337959424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed75701517564416 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 % _timestamp1.7884678942472591e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed350263300708864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed18387156 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated22875148 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed350263300708864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated18387627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3693108537615 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed11322046 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated29973179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated346565363387392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3700479112192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated332502429392944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed336195512502591 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4731824651264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated336198052298967 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3700479112192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed346565337959424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 trainer/global_step1599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated29973179 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3697962749440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated52848327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated22875148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current471 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current468 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated70969722011136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 _step127 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed22875142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed29953496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed29953496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7065113 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4731883591168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed346565337959424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed75701517564416 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 % _timestamp1.7884678942472591e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed350263300708864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed18387156 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated22875148 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed350263300708864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated18387627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3693108537615 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed11322046 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated29973179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated346565363387392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3700479112192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated332502429392944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed336195512502591 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4731824651264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated336198052298967 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3700479112192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed346565337959424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 trainer/global_step1599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated29973179 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3697962749440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated52848327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated22875148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current471 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current468 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated70969722011136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated346776353762816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3700237965824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated22889029 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated22889029 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _runtime2785.855277197 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed52860870 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4734734007808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed52860870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated52880585 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed22889023 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed11328898 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated52880585 WCDeviceStatsMonitor.on_train_batch_end/active_bNytes.large_pool.freed346776328334848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated346776353762816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed29971847 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 % _timestamp1.7884678958129125e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3700237965824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7069442 _step128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4734792934400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed350476566300672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated350479108104704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated29991556 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed18398340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed71013754282496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated11329367 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed29971847 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3695380792469 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3697895161085 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146#' M UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3695380792469 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3697895161085 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed22889023 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed332704932674560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated332704958102528 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated7069445 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated346776353762816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3700237965824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated22889029 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated22889029 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _runtime2785.855277197 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed52860870 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4734734007808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed52860870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated52880585 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed22889023 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed11328898 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated52880585 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed346776328334848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated346776353762816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed29971847 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 % _timestamp1.7884678958129125e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3700237965824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7069442 _step128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 4734792934400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed350476566300672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated350479108104704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated29991556 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed18398340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed71013754282496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated11329367 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed29971847g2 - _wandbruntime2785  _runtime2785X  2loss/SRqqt3/assignment_loss1.7172193318082576 0loss/SRqqt4/assignment_loss1.79054545451048 1loss/FBt1/assignment_loss0.14444783250470059 /loss/FBt4/detection_loss2.7203133137602555 _step129 /loss/FRt1/detection_loss1.9425039491232703 /loss/FRt3/detection_loss3.0906969511284017 /loss/FBt1/detection_loss0.5790300858660994 1loss/FBt3/assignment_loss0.42073531878607656 .loss/FRt4/detection_loss2.169497886300087 0loss/SRqqt3/detection_loss2.533479776298791 2loss/SRqqt2/assignment_loss1.8059021902305108 /loss/FRt2/detection_loss2.3046163255103456 /loss/SRqqt2/detection_loss2.24751349253419 epoch2 trainer/global_step1599 0loss/FBt4/assignment_loss0.3251657318155355 $loss/total_loss45.9206740656642 % _timestamp1.7884678958171241e+09 0loss/FRt3/assignment_loss3.8277317375506996 2loss/SRqqt1/assignment_loss1.0537545265974821 /loss/FBt3/detection_loss3.3780928403139114 0loss/FRt1/assignment_loss1.5985243586932911 0loss/FRt2/assignment_loss3.8671153627690815 0loss/FBt2/assignment_loss0.2589225272405083 1loss/SRqqt1/detection_loss1.0061933903230562 1loss/SRqqt4/detection_loss1.7059077928381519 _runtime2785.857584508 /loss/FRt4/assignment_loss3.199532940712842 .loss/FBt2/detection_loss2.233230948448181E]  2loss/SRqqt2/assignment_loss1.8059021902305108 /loss/FRt2/detection_loss2.3046163255103456 /loss/SRqqt2/detection_loss2.24751349253419 epoch2 trainer/global_step1599 0loss/FBt4/assignment_loss0.3251657318155355 $loss/total_loss45.9206740656642 % _timestamp1.7884678958171241e+09 0loss/FRt3/assignment_loss3.8277317375506996 2loss/SRqqt1/assignment_loss1.0537545265974821 /loss/FBt3/detection_loss3.3780928403139114 0loss/FRt1/assignment_loss1.5985243586932911 0loss/FRt2/assignment_loss3.8671153627690815 0loss/FBt2/assignment_loss0.2589225272405083 1loss/SRqqt1/detection_loss1.0061933903230562 1loss/SRqqt4/detection_loss1.7059077928381519 _runtime2785.857584508 /loss/FRt4/assignment_loss3.199532940712842 .loss/FBt2/detection_loss2.233230948448181 2loss/SRqqt3/assignment_loss1.7172193318082576 0loss/SRqqt4/assignment_loss1.79054545451048 1loss/FBt1/assignment_loss0.14444783250470059 /loss/FBt4/detection_loss2.7203133137602555 _step129 /loss/FRt1/detection_loss1.9425039491232703 /loss/FRt3/detection_loss3.0906969511284017 /loss/FBt1/detection_loss0.5790300858660994 1loss/FBt3/assignment_loss0.42073531878607656 .loss/FRt4/detection_loss2.169497886300087 0loss/SRqqt3/detection_loss2.533479776298791X! (:  gpu.0.gpu46.0 gpu.0.memory11+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts148.264' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent29.652800000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0mu (: ! disk.nvme0n1.in 6350.96484375& disk.nvme0n1.out62759.34130859375* proc.memory.percent19.072723388671875# disk.dm-0.out62759.34130859375 disk.sda1.in0 disk.sda2.in0( disk.nvme0c0n1.out62759.34130859375 disk.dm-0.in 6350.96484375 disk.sda3.out 85.58984375 disk.sda.in0 disk.dm-1.in0 disk.dm-1.out 85.58984375 disk.drbd1260.out 0.3828125 cpu25.109169647550818 proc.cpu.threads17 network.sent 3.495132e+06 network.recv887144 disk.sda1.out0 disk.sda.out 85.58984375 disk.drbd1260.in0$ memory_percent88.96088600158691( proc.memory.availableMB 1808.6484375 proc.memory.rssMB3124.875) disk./.usagePercent77.17023884560838% disk./.usageGB1149.8923263549805 disk.sda3.in0 disk.sda2.out0# disk.nvme0c0n1.in 6350.96484375] (: ʵ gpu.0.gpu98.0 gpu.0.memory23+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts260.202' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent52.0404 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0'v (: ʵ% disk./.usageGB1149.8923034667969 disk.sda1.in0 disk.sda.in0# disk.nvme0c0n1.in 6350.96484375 disk.drbd1260.out 0.3828125$ memory_percent88.95938396453857 cpu25.111059394217087 proc.cpu.threads17! disk.nvme0n1.in 6350.96484375& disk.nvme0n1.out62854.66943359375 disk.sda3.out 86.06640625 disk.sda.out 86.06640625 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.drbd1260.in0) disk./.usagePercent77.17023730956316 disk.dm-0.in 6350.96484375# disk.dm-0.out62854.66943359375 disk.sda1.out0 disk.sda3.in0( disk.nvme0c0n1.out62854.66943359375 disk.dm-1.out 86.06640625 proc.memory.rssMB3124.875 network.sent 3.581482e+06 network.recv891978) proc.memory.availableMB 1808.89453125* proc.memory.percent19.072723388671875:] (: ٵ# memory_percent88.9594554901123! disk.nvme0n1.in 6503.08984375 disk.dm-0.in 6503.08984375 disk.sda1.out0 disk.sda3.in0# disk.nvme0c0n1.in 6503.08984375 disk.dm-1.in0( proc.memory.availableMB 1808.8828125 proc.memory.rssMB3124.875 network.sent 3.589315e+06 network.recv896760 disk.dm-0.out 63226.1875 disk.sda3.out 86.56640625 disk.sda2.out0 disk.dm-1.out 86.56640625* proc.memory.percent19.072723388671875) disk./.usagePercent77.17706503050135$ disk./.usageGB1149.994041442871 disk.drbd1260.out0.390625 disk.nvme0n1.out 63226.1875! disk.nvme0c0n1.out 63226.1875 cpu25.112884424596555 proc.cpu.threads17 disk.drbd1260.in0 disk.sda1.in0 disk.sda.in0 disk.sda.out 86.56640625 disk.sda2.in0S (: ٵ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts129.863' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent25.9726 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0o, (:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts338.354' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.6708 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0h (: 挪 disk.sda1.out0 disk.sda3.out 86.96484375 disk.sda.in0 disk.sda2.in0 disk.sda2.out0 network.sent 3.592916e+06 network.recv900978 disk.drbd1260.in0 disk.drbd1260.out0.390625 disk.dm-0.in 6503.08984375 disk.dm-0.out63572.36328125% disk.nvme0c0n1.out63572.36328125$ memory_percent88.95940780639648! disk.nvme0n1.in 6503.08984375# disk.nvme0n1.out63572.36328125 disk.sda.out 86.96484375# disk.nvme0c0n1.in 6503.08984375 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875 cpu25.11497984358706) disk./.usagePercent77.18547206191828$ disk./.usageGB1150.119312286377 disk.sda3.in0' proc.memory.availableMB 1808.890625 proc.cpu.threads17 disk.dm-1.in0 disk.dm-1.out 86.96484375 disk.sda1.in0 (:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts248.234' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.6468 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0x,b(: ( disk.nvme0c0n1.out63979.51220703125 disk.dm-1.in0 network.recv904750 disk.dm-0.in 6503.08984375 disk.sda2.out0# disk.nvme0c0n1.in 6503.08984375 disk.dm-1.out87.375 disk.drbd1260.in0$ memory_percent88.95940780639648' proc.memory.availableMB 1808.890625$ disk./.usageGB1149.871295928955# disk.dm-0.out63979.51220703125 disk.sda1.out0 disk.sda3.in0 disk.drbd1260.out0.390625 proc.memory.rssMB3124.875 cpu25.116719311500706& disk.nvme0n1.out63979.51220703125 disk.sda3.out87.375 disk.sda.out87.375* proc.memory.percent19.072723388671875 proc.cpu.threads17 disk.sda.in0 network.sent 3.598573e+06( disk./.usagePercent77.1688274760721! disk.nvme0n1.in 6503.08984375 disk.sda1.in0 disk.sda2.in0t޾(: ډ gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp50.0 gpu.0.powerWatts336.0' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.2 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0wp(:  network.sent 3.603588e+06! disk.nvme0n1.in 6503.08984375 disk.dm-0.in 6503.08984375 disk.sda1.in0 disk.sda2.in0# disk.nvme0c0n1.in 6503.08984375 disk.drbd1260.out 0.39453125 proc.memory.rssMB3124.875 network.recv908518 disk.sda.in0 disk.sda2.out0 disk.dm-1.in0) proc.memory.availableMB 1808.88671875 disk.sda.out 87.890625) disk./.usagePercent77.13423957815857% disk./.usageGB1149.3559112548828# disk.dm-0.out64489.05517578125 disk.sda3.out 87.890625 disk.dm-1.out 87.890625* proc.memory.percent19.072723388671875 cpu25.118710509071416& disk.nvme0n1.out64489.05517578125 disk.sda1.out0 disk.sda3.in0( disk.nvme0c0n1.out64489.05517578125 disk.drbd1260.in0# memory_percent88.9594316482544 proc.cpu.threads17ʎ2- _wandbruntime2866  _runtime2866 & lr-AdamW/pg10.0010735873725943022 & lr-AdamW/pg20.0010735873725943022 trainer/global_step1649 $ _timestamp1.788467976406231e+09 _runtime2866.446791859 _step130If3 & lr-AdamW/pg20.0010735873725943022 trainer/global_step1649 $ _timestamp1.788467976406231e+09 _runtime2866.446791859 _step130 & lr-AdamW/pg10.0010735873725943022oqaQ2- _wandbruntime2866  _runtime2866{O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated23569198 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3809262820731 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 trainer/global_step1649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated30894081 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3811751396864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed30874398 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current461 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3814267759616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7281713 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 $ _timestamp1.788467976407594e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed18950560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4877365089792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 _step131 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed54443590 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed73172761380352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed360926608127488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed23569192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated342628864872144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed360926608127488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed357114856730624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3811751396864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed11668850 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated78050212410880 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated357114882158592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed346435587896499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 VGDeviceStatsMoni6wtor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated54463279 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated30894081 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated360929149918208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated23569198 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed78050126470144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated357114882158592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated346438127692875 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated18951024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed342628839444176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3806748452323 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated360929149918208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4877421932544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed23569192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3814267759616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7281710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated73172790478336 _runtime2866.448410601d'O F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated73172790478336 _runtime2866.448410601 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated23569198 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 3809262820731 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 trainer/global_step1649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated30894081 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3811751396864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed30874398 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current461 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 3814267759616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7281713 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 $ _timestamp1.788467976407594e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed18950560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 4877365089792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 _step131 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed54443590 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed73172761380352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed360926608127488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed23569192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated342628864872144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed360926608127488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed357114856730624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 3811751396864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed11668850 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated78050212410880 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated357114882158592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed346435587896499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated54463279 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated30894081 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated360929149918208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated23569198 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed78050126470144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated357114882158592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated346438127692875 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated18951024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed342628839444176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3806748452323 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated360929149918208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 4877421932544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed23569192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3814267759616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7281710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248Z֌2- _wandbruntime2867  _runtime2867d&M _runtime 2867.95863343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed30892749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed18961802 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated73216851847680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed30892749 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated54495537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated18962263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated357325872534016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85927424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated361142415523328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3816542989312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed346640388860937 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated54495537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated11676218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed78097102542848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step1649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4880279793152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated361142415523328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed54475822 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated78097188470272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated357325872534016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3809020707177 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed54475822 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed342831368153760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3811535075793 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed23583073 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated342831393581728pCAM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 3816542989312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed361139873719296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed357325847106048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed73216822749696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3814026613248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current461 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7286042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated18962263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated357325872534016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85927424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated361142415523328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3816542989312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed346640388860937 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated54495537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated11676218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed78097102542848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step1649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 4880279793152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated361142415523328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed54475822 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated78097188470272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56829440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated3573258I 72534016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3809020707177 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed54475822 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed342831368153760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3811535075793 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed23583073 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated342831393581728 _runtime 2867.95863343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed30892749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed18961802 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated73216851847680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed30892749 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated54495537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3925540044288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed19514136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3920388367031 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed371589915546112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated56078231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed31795300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3928056407040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7498310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated356678203086783 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed24263242 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated352755300351344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated367664400929792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated371592457336832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated31814983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed80398867255296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated367664400929792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed367664375501824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed371589915546112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated56078231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed31795300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 3928056407040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7498310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated356678203086783 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed24263242 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated352755300351344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated367664400929792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated371592457336832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated31814983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed80398867255296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated367664400929792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed367664375501824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 3925540044288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.Afreed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed19514136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 3920388367031 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980lj52- _wandbruntime2947  _runtime2947h&M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated24277129 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed19525368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 _runtime2947.241075581 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated56110489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current477 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated7502645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 trainer/global_step1699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed31813651 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3925174990501 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated371805722941952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed371803181137920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12022726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated31833360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed367875365877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated31833360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3922660621885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3927815260672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5025945719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed352957803632960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _step136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated24277129 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 $ _timestamp1.788468057199825e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3927815260672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current474 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed75419891216896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed31813651 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3930331636736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated75419920314880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7502642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated352957829060928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed56090774 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated367875391305216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed24277123 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated356883004051429\yl'M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed56090774 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated367875391305216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed24277123 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated356883004051429 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated24277129 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed19525368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 _runtime2947.241075581 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated56110489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current477 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated7502645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 trainer/global_step1699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed31813651 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 3925174990501 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated371805722941952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed371803181137920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12022726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19715 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated31833360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed367875365877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated31833360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 3922660621885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 3927815260672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5025945719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed352957803632960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _step136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated24277129 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 $ _timestamp1.788468057199825e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 3927815260672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current474 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed75419891216896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed31813651 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 3930331636736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated75419920314880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7502642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated352957829060928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current692- _wandbruntime2947  _runtime29475]c=V /loss/FRt1/assignment_loss1.540982080760755 2loss/SRqqt1/assignment_loss1.2319761620944036 2loss/SRqqt3/assignment_loss1.7606731534406945 1loss/SRqqt4/assignment_loss1.760200487544476 1loss/FBt4/assignment_loss0.31423633673512086 .loss/FBt1/detection_loss0.609705321362936 .loss/FRt2/assignment_loss3.89104155699412 1loss/FBt2/assignment_loss0.23007634180513295 epoch2 trainer/global_step1699 _runtime 2947.24296287 /loss/FRt2/detection_loss2.5957041831984036 0loss/SRqqt3/detection_loss2.488203367268717 1loss/SRqqt4/detection_loss1.7727581430542267 %loss/total_loss44.10914340781088 " _timestamp1.7884680572025e+09 0loss/FRt3/assignment_loss3.6077837239612234 /loss/FRt4/detection_loss2.1649763574739835 1loss/SRqqt2/detection_loss2.1707006258729065 .loss/FBt3/detection_loss2.526278575261434 0loss/FBt3/assignment_loss0.2613823772155835 1loss/SRqqt1/detection_loss0.9223844256568579 2loss/SRqqt2/assignment_loss1.6364274147300073 1loss/FBt1/assignment_loss0.12272928675082642 /loss/FRt1/detection_loss1.7092543587872857 _step137 /loss/FRt4/assignment_loss3.092211206754049 /loss/FRt3/detection_loss2.8096134716814216 /loss/FBt2/detection_loss2.7485529726201836 /loss/FBt4/detection_loss2.1412914767861366DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed382253222964736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4039328691712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed12363216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed24957292 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current468 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated32735885 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated362881735830544 ]Y:IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated378213919700992 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed57673494 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed378213894273024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated24957298 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed32716202 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed82747381856256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed77578898314752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated57693183 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4039328691712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4034028281739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed32716202 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4036542650147 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated20078597 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed24957292 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4041845054464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed20078126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated366918278480691 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed378213894273024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated378213919700992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated382255764755456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7714913 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated82747469894144 % _timestamp1.7884681383267205e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated24957298 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated12363684 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5168542481408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461'O R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed382253222964736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4039328691712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed12363216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed24957292 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current468 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated32735885 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated362881735830544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated378213919700992 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed57673494 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed378213894273024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated24957298 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed32716202 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed82747381856256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed77578898314752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated57693183 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4039328691712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142752768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4034028281739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed32716202 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4036542650147 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated20078597 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed24957292 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4041845054464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed20078126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated366918278480691 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed378213894273024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated378213919700992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated382255764755456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7714913 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516362752 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated82747469894144 % _timestamp1.7884681383267205e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated24957298 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated12363684 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5168542481408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 369380114612cg2- _wandbruntime3029  _runtime3029z &M WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated382469030360576 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed77622959684096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 _runtime3029.868489337 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed382466488556544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current459 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12370069 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed82794349294592 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed378424884648448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated378424910076416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed20089311 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated57725441 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed57705726 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 $ _timestamp1.788468139827353e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated12370528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed367120539648753 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7719242 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 trainer/global_step1749 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed24971173 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed378424884648448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4041603908096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed57705726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step140 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated82794437319168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated363084264540128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed363084239112160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4036300536593 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated24971179 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated77622988782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated382469030360576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated57725441 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated20089773 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed24971173 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5171389610496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146ʣM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated382469030360576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142752768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated57725441 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated20089773 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed24971173 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5171389610496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated382469030360576 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed77622959684096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 _runtime3029.868489337 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed382466488556544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current459 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12370069 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed82794349294592 Pytes.small_pool.freed0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23980 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23681 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4038814905209 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed378424884648448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated378424910076416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed20089311 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated57725441 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19709 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed57705726 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 $ _timestamp1.788468139827353e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated12370528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed367120539648753 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7719242 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 trainer/global_step1749 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed24971173 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed378424884648448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4041603908096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed57705726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 _step140 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated82794437319168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142752768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated363084264540128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed363084239112160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4036300536593 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated24971179 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated77622988782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527Y/2- _wandbruntime3029  _runtime3029VY epoch2 1loss/FBt1/assignment_loss0.15000503073227137 0loss/FBt2/assignment_loss0.2845530530263204 /loss/FBt3/detection_loss2.6975853192178825 0loss/FRt1/assignment_loss1.4197663037727277 0loss/FRt4/assignment_loss3.1000471166056447 1loss/SRqqt4/assignment_loss1.677498561101702 .loss/FRt2/detection_loss2.565505928652627 1loss/SRqqt1/detection_loss1.0799068493768573 1loss/SRqqt4/detection_loss1.7911423834146194 trainer/global_step1749 _runtime3029.870669268 1loss/SRqqt3/assignment_loss1.683850050168602 0loss/SRqqt2/detection_loss2.070234810312589 /loss/FBt1/detection_loss0.6861650425849891 /loss/FRt3/assignment_loss3.420627255524908 2loss/SRqqt1/assignment_loss1.2320841674692928 2loss/SRqqt2/assignment_loss1.7089041440851158 /loss/FRt1/detection_loss1.8495228262618184 .loss/FRt3/detection_loss2.943457694990294 1loss/SRqqt3/detection_loss2.3890175333148553 /loss/FRt2/assignment_loss3.379521288190569 .loss/FBt4/detection_loss2.267305658923255 %loss/total_loss43.22257100787495 % _timestamp1.7884681398297675e+09 _step141 1loss/FBt3/assignment_loss0.20250493307915013 1loss/FBt4/assignment_loss0.19062103745656916 .loss/FRt4/detection_loss1.979168286308264 /loss/FBt2/detection_loss2.4535757333040236:!^ .loss/FRt3/detection_loss2.943457694990294 1loss/SRqqt3/detection_loss2.3890175333148553 /loss/FRt2/assignment_loss3.379521288190569 .loss/FBt4/detection_loss2.267305658923255 %loss/total_loss43.22257100787495 % _timestamp1.7884681398297675e+09 _step141 1loss/FBt3/assignment_loss0.20250493307915013 1loss/FBt4/assignment_loss0.19062103745656916 .loss/FRt4/detection_loss1.979168286308264 /loss/FBt2/detection_loss2.4535757333040236 epoch2 1loss/FBt1/assignment_loss0.15000503073227137 0loss/FBt2/assignment_loss0.2845530530263204 /loss/FBt3/detection_loss2.6975853192178825 0loss/FRt1/assignment_loss1.4197663037727277 0loss/FRt4/assignment_loss3.1000471166056447 1loss/SRqqt4/assignment_loss1.677498561101702 .loss/FRt2/detection_loss2.565505928652627 1loss/SRqqt1/detection_loss1.0799068493768573 1loss/SRqqt4/detection_loss1.7911423834146194 trainer/global_step1749 _runtime3029.870669268 1loss/SRqqt3/assignment_loss1.683850050168602 0loss/SRqqt2/detection_loss2.070234810312589 /loss/FBt1/detection_loss0.6861650425849891 /loss/FRt3/assignment_loss3.420627255524908 2loss/SRqqt1/assignment_loss1.2320841674692928 2loss/SRqqt2/assignment_loss1.7089041440851158 /loss/FRt1/detection_loss1.8495228262618184(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts270.08' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent54.016 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ťt(:  disk.sda3.out 92.76953125 disk.sda2.in0( disk.nvme0c0n1.out67709.98876953125) proc.memory.availableMB 1808.82421875 proc.memory.rssMB3124.875 proc.cpu.threads17 disk.sda1.in0 disk.sda3.in0 network.sent 3.840238e+06 network.recv956674) disk./.usagePercent77.08536389975909& disk.nvme0n1.out67709.98876953125 disk.dm-0.in 6918.23828125# disk.dm-0.out67709.98876953125$ disk./.usageGB1148.627628326416 disk.drbd1260.out 0.4140625 disk.sda1.out0 disk.sda.in0 disk.sda.out 92.76953125* proc.memory.percent19.072723388671875 disk.drbd1260.in0! disk.nvme0n1.in 6918.23828125 disk.sda2.out0# disk.nvme0c0n1.in 6918.23828125 disk.dm-1.in0 disk.dm-1.out 92.76953125$ memory_percent88.95981311798096 cpu25.13777459423702&(: Έ gpu.0.gpu74.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts225.766' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.1532 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0)Gq(:  network.recv961857% disk./.usageGB1148.8566436767578& disk.nvme0n1.out68079.51416015625 disk.sda1.out0 disk.dm-1.in0* proc.memory.percent19.072723388671875 cpu25.13927355330489( disk./.usagePercent77.1007333120772 disk.sda.out 93.18359375 disk.sda2.out0 disk.drbd1260.in0 disk.drbd1260.out 0.41796875$ memory_percent88.95988464355469% proc.memory.availableMB 1808.8125! disk.nvme0n1.in 6918.23828125# disk.dm-0.out68079.51416015625 disk.sda3.out 93.18359375 disk.sda.in0# disk.nvme0c0n1.in 6918.23828125 proc.cpu.threads17 disk.dm-0.in 6918.23828125 disk.sda1.in0 disk.sda3.in0 disk.sda2.in0( disk.nvme0c0n1.out68079.51416015625 disk.dm-1.out 93.18359375 proc.memory.rssMB3124.875 network.sent 3.913551e+06o(: ɷ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts235.39' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.077999999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors06t(: ɷ disk.sda2.out0# memory_percent88.9613389968872 disk.sda.out 93.62890625( disk.nvme0c0n1.out68174.84228515625 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875 proc.cpu.threads17) disk./.usagePercent77.10192425912649& disk.nvme0n1.out68174.84228515625 disk.sda1.out0 disk.sda3.out 93.62890625 disk.sda.in0 disk.sda2.in0) proc.memory.availableMB 1808.57421875 cpu25.1408291823232 network.recv966613 disk.drbd1260.in0 disk.dm-0.in 6918.23828125 disk.sda1.in0# disk.nvme0c0n1.in 6918.23828125 disk.dm-1.in0 disk.dm-1.out 93.62890625 network.sent 3.919528e+06% disk./.usageGB1148.8743896484375 disk.drbd1260.out 0.41796875! disk.nvme0n1.in 6918.23828125# disk.dm-0.out68174.84228515625 disk.sda3.in0;5^(: طќ$ disk./.usageGB1148.873878479004 disk.drbd1260.in0 disk.nvme0n1.in 7062.6875# disk.nvme0n1.out68709.86328125 disk.sda2.in0 disk.sda2.out0% disk.nvme0c0n1.out68709.86328125 disk.dm-1.out 94.02734375 disk.drbd1260.out0.421875 disk.dm-0.out68709.86328125 disk.sda.in0$ memory_percent88.96143436431885) proc.memory.availableMB 1808.55859375* proc.memory.percent19.072723388671875 proc.cpu.threads17 disk.dm-0.in 7062.6875 disk.sda1.out0 proc.memory.rssMB3124.875 cpu25.142550725125624 network.recv970621( disk./.usagePercent77.1018899541169 disk.dm-1.in0 disk.sda1.in0 disk.sda3.in0 disk.sda3.out 94.02734375 disk.sda.out 94.02734375 disk.nvme0c0n1.in 7062.6875 network.sent 3.925238e+06 ~=(: ط gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts243.14' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.628 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Qn>(: և gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts221.439' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent44.2878 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0B QP(: % proc.memory.availableMB 1808.8125 proc.memory.rssMB3124.875* proc.memory.percent19.072723388671875 proc.cpu.threads17 network.sent 3.930778e+06) disk./.usagePercent77.10188637001141 disk.nvme0c0n1.in 7062.6875 disk.dm-1.in0 disk.drbd1260.out0.421875 disk.dm-0.in 7062.6875 disk.sda3.out 94.5390625 disk.sda.in0% disk./.usageGB1148.8738250732422 disk.dm-1.out 94.5390625 disk.drbd1260.in0 disk.nvme0n1.out 68878.34375 disk.dm-0.out 68878.34375 disk.sda3.in0 disk.sda.out 94.5390625 disk.sda2.in0" disk.nvme0c0n1.out 68878.34375 disk.nvme0n1.in 7062.6875 disk.sda1.in0 disk.sda1.out0 disk.sda2.out0 cpu25.144231933714096 network.recv974604$ memory_percent88.95988464355469ϴ`(:  network.recv977941% disk.nvme0c0n1.out69167.48046875 disk.drbd1260.in0 disk.sda2.in0 proc.memory.rssMB3124.875 cpu25.145934368451435 disk.dm-1.out 94.96484375 disk.sda.in0# memory_percent88.9585018157959* proc.memory.percent19.072723388671875 proc.cpu.threads17 disk.nvme0n1.in 7062.6875 disk.dm-0.in 7062.6875 network.sent 3.937692e+06) disk./.usagePercent77.09353514823697 disk.nvme0c0n1.in 7062.6875 disk.dm-1.in0 disk.drbd1260.out 0.42578125 disk.sda1.in0% disk./.usageGB1148.7493858337402# disk.nvme0n1.out69167.48046875 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 94.96484375 disk.sda.out 94.96484375 disk.sda2.out0( proc.memory.availableMB 1809.0390625 disk.dm-0.out69167.48046875o(: ꑿ gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts190.437' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent38.0874 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors02- _wandbruntime3110  _runtime3110I _step142 & lr-AdamW/pg10.0010428138049824055 & lr-AdamW/pg20.0010428138049824055 trainer/global_step1799 $ _timestamp1.788468220433999e+09 _runtime3110.474788886b7  & lr-AdamW/pg10.0010428138049824055 & lr-AdamW/pg20.0010428138049824055 trainer/global_step1799 $ _timestamp1.788468220433999e+09 _runtime3110.474788886 _step142H f2- _wandbruntime3110  _runtime3110uC'O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed33637104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4147668196447 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed373008145881776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated79781995879936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed79781966781952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 trainer/global_step1799 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4150182564855 % _timestamp1.7884682204359949e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed33637104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated392919072174080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated388763438472192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated25651348 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated85095985227264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 _step143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4155633701888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated20642186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated12710673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed388763413044224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4155633701888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated373008171309744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed25651342 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5313989347328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5313930407424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated388763438472192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed12710209 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7931513 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated33656787 _runtime3110.476986279 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4153117339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated392919072174080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4153117339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed20641719 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7931510 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated377158353874599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed59288446 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated59308135 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated25651348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed388763413044224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed392916530383360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed377155814078223HxO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated33656787 _runtime3110.476986279 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19689 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4153117339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated392919072174080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23980 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4153117339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed20641719 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed7931510 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated377158353874599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed59288446 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated59308135 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated25651348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516362752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142752768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed388763413044224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541790720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed392916530383360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed377155814078223 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736372 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed33637104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4147668196447 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23681 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed373008145881776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541790720 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated79781995879936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed79781966781952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 trainer/global_step1799 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011461 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4150182564855 % _timestamp1.7884682204359949e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19683 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19683 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed33637104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated392919072174080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated388763438472192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated25651348 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated85095985227264 GADeviceStatsMonitor.on_train_batch_start/oversize_a Wπllocations.peak0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 _step143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4155633701888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated20642186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated12710673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed388763413044224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23681 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4155633701888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated373008171309744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed25651342 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5313989347328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5313930407424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated388763438472192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed12710209 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated7931513 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed59320678 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated33675164 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4155392555520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed33655455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current463 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed20652875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed388974403419648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated33675164 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12717033 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 trainer/global_step1799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed33655455 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5316892661760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated25665229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated59340393 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 _step144 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4157908931584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated20653338 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7935842 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated85142949911040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed25665223 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed79826028151296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated59340393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime3112.039346824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated388974428847616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed59320678 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated33675164 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4155392555520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed33655455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current463 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011461 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed20652875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed388974403419648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated33675164 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed12717033 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 trainer/global_step1799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed33655455 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5316892661760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated25665229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368616 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated59340393 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736372 _step144 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4157908931584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated20653338 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed7935842 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23980 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated85142949911040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed25665223 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed79826028151296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated59340393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19709 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _runtime3112.039346824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated388974428847616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 F&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 & Purity/3FRt1SRqqt1FBt/FBt_purity1 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.24814090132713318 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 ) Purity/*FRt4SRqqt*FBt/FRt_purity0.25 ) Purity/0FRt0SRqqt4FBt/FBt_purity0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity0 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.3060974180698395 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ("Purity/4FRt2SRqqt*FBt/event_purity0 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32670044898986816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.4103369414806366 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9758555293083191 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 & Purity/1FRt*SRqqt4FBt/FRt_purity0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.2670695185661316 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.3333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 :"Purity/*FRt*SRqqt3FBt/event_purity0.09195402264595032 8 Purity/1FRt0SRqqt*FBt/FRt_purity0.25639331340789795 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.25460830330848694 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.38945698738098145 ("Purity/2FRt2SRqqt*FBt/event_purity0 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.4041556417942047 ("Purity/0FRt0SRqqt0FBt/event_purity1 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.25935959815979004 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.22769273817539215 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 +jet/accuracy_3_of_30.01847144216299057 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.36626383662223816 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.22991222143173218 ("Purity/2FRt2SRqqt0FBt/event_purity0 ("Purity/3FRt0SRqqt0FBt/event_purity0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.21195651590824127 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 particle/accuracy_3_of_41 jet/accuracy_5_of_60 8"Purity/1FRt0SRqqt1FBt/event_purity0.239681676030159 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.408525288105011 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 particle/accuracy_4_of_71 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 & Purity/*FRt0SRqqt4FBt/FRt_purity0 & Purity/1FRt4SRqqt0FBt/FRt_purity0 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 9"Purity/0FRt0SRqqt2FBt/event_purity0.3033808469772339 , Purity/1FRt3SRqqt*FBt/FBt_purity0.96875 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.2217460423707962 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 particle/accuracy_3_of_51 & Purity/0FRt4SRqqt*FBt/FBt_purity1 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 9"Purity/0FRt0SRqqt1FBt/event_purity0.9993136525154114 :"Purity/0FRt2SRqqt0FBt/event_purity0.02492099069058895 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.25453150272369385 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.125 6 Purity/0FRt*SRqqt*FBt/FBt_purity0.983864426612854 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 particle/accuracy_3_of_31 & Purity/4FRt0SRqqt*FBt/FRt_purity0 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9945845007896423 8 Purity/1FRt*SRqqt2FBt/FRt_purity0.49648764729499817 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.22720058262348175 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 8 Purity/1FRt1SRqqt*FBt/FRt_purity0.25244107842445374 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 & Purity/*FRt1SRqqt4FBt/FRt_purity0 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 7 Purity/2FRt1SRqqt1FBt/FRt_purity0.3188747763633728 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 particle/accuracy_6_of_71 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 :"Purity/0FRt*SRqqt3FBt/event_purity0.12345678359270096 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 8 Purity/1FRt1SRqqt1FBt/FRt_purity0.29422077536582947 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.4037037491798401 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.24702021479606628 7 Purity/*FRt0SRqqt2FBt/FRt_purity0.4928172826766968 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 7 Purity/2FRt1SRqqt*FBt/FRt_purity0.2510954737663269 8 Purity/2FRt3SRqqt0FBt/FRt_purity0.31617963314056396 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 particle/accuracy_4_of_41 & Purity/2FRt3SRqqt*FBt/FBt_purity1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity0.125 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 ("Purity/2FRt3SRqqt*FBt/event_purity0 *jet/accuracy_2_of_40.2889387011528015 +jet/accuracy_3_of_40.04575696960091591 ("Purity/1FRt4SRqqt*FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FRt_purity1 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 8 Purity/2FRt*SRqqt1FBt/FRt_purity0.32143744826316833 ,jet/accuracy_4_of_50.007857734337449074 & Purity/2FRt3SRqqt1FBt/FBt_purity1 _runtime3184.338633593 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 +jet/accuracy_2_of_30.19193962216377258 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.2300773411989212 8 Purity/2FRt*SRqqt*FBt/FRt_purity0.24266502261161804 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.6754385828971863 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8492063879966736 ;"Purity/*FRt1SRqqt2FBt/event_purity0.033591676503419876 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9867276549339294 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9958243370056152 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 particle/accuracy_2_of_51 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.261904776096344 _step146 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.21345369517803192 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ("Purity/3FRt1SRqqt1FBt/event_purity0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 & Purity/0FRt4SRqqt1FBt/FBt_purity1 9"Purity/*FRt0SRqqt1FBt/event_purity0.6746923327445984 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 ( Purity/2FRt4SRqqt*FBt/FRt_purity0.5 ("Purity/0FRt1SRqqt3FBt/event_purity0 :"Purity/*FRt*SRqqt2FBt/event_purity0.16256065666675568 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.9113095998764038 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.2278536856174469 ("Purity/2FRt1SRqqt*FBt/event_purity0 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity 0.1953125 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 +jet/accuracy_2_of_60.28947368264198303 ("Purity/1FRt1SRqqt3FBt/event_purity0 9"Purity/*FRt*SRqqt*FBt/event_purity0.1882375031709671 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 particle/accuracy_1_of_61 jet/accuracy_1_of_71 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9723148345947266 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.27283933758735657 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 -jet/accuracy_4_of_40.0010080644860863686 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.9270833134651184 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 :"Purity/3FRt*SRqqt*FBt/event_purity0.00358422938734293 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt0SRqqt2FBt/event_purity0 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.22511614859104156 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 ("Purity/*FRt2SRqqt3FBt/event_purity0 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 ;"Purity/1FRt1SRqqt0FBt/event_purity0.008387289941310883 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 :"Purity/0FRt1SRqqt2FBt/event_purity0.06296297162771225 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.31481480598449707 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 jet/accuracy_5_of_50 7validation_average_jet_accuracy0.34588029980659485 ("Purity/0FRt4SRqqt*FBt/event_purity0 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.3333333432674408 5 Purity/*FRt*SRqqt0FBt/FRt_purity0.20804363489151 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.38657405972480774 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ) Purity/*FRt4SRqqt0FBt/FRt_purity0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity0 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9868655800819397 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.39307278394699097 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 6 Purity/*FRt0SRqqt4FBt/FBt_purity0.800000011920929 & Purity/*FRt4SRqqt*FBt/FBt_purity1 ("Purity/1FRt2SRqqt2FBt/event_purity0 epoch2 7 Purity/1FRt2SRqqt0FBt/FRt_purity0.2162618190050125 ("Purity/1FRt3SRqqt0FBt/event_purity0 ("Purity/2FRt1SRqqt2FBt/event_purity0 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 :"Purity/3FRt0SRqqt1FBt/event_purity0.03846153989434242 & Purity/1FRt4SRqqt*FBt/FRt_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ("Purity/2FRt2SRqqt2FBt/event_purity0 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9860703945159912 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.21237719058990479 ("Purity/3FRt3SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.3924858868122101 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.17885248363018036 ("Purity/*FRt3SRqqt1FBt/event_purity0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4327944815158844 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9907825589179993 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9761905074119568 ("Purity/1FRt2SRqqt1FBt/event_purity0 jet/accuracy_3_of_70 jet/accuracy_7_of_70 * Purity/1FRt*SRqqt4FBt/FBt_purity0.875 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 & Purity/4FRt0SRqqt0FBt/FRt_purity0 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.20000000298023224 +jet/accuracy_2_of_20.08624794334173203 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.3333333432674408 particle/accuracy_4_of_61 & Purity/2FRt0SRqqt3FBt/FBt_purity1 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.2811186909675598 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.2664150893688202 :"Purity/1FRt*SRqqt0FBt/event_purity0.07845912128686905 ("Purity/*FRt4SRqqt0FBt/event_purity0 6 Purity/*FRt*SRqqt2FBt/FBt_purity0.978058397769928 ( Purity/*FRt0SRqqt3FBt/FRt_purity0.5 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.3935227692127228 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt3SRqqt*FBt/FRt_purity0.2473374307155609 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/3FRt4SRqqt*FBt/event_purity0 :"Purity/*FRt*SRqqt0FBt/event_purity0.10097857564687729 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 ;"Purity/2FRt0SRqqt0FBt/event_purity0.010075408965349197 ;"Purity/2FRt0SRqqt1FBt/event_purity0.027037037536501884 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 8 Purity/2FRt1SRqqt0FBt/FRt_purity0.22262871265411377 7 Purity/2FRt3SRqqt*FBt/FRt_purity0.3014761805534363 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 +jet/accuracy_3_of_60.05263157933950424 particle/accuracy_3_of_61 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.43217846751213074 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.2937720715999603 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.23262843489646912 8 Purity/*FRt2SRqqt2FBt/FRt_purity0.44871795177459717 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9791666865348816 particle/accuracy_7_of_71 ("Purity/0FRt2SRqqt3FBt/event_purity0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.41228678822517395 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 , Purity/1FRt*SRqqt3FBt/FRt_purity0.53125 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9849491119384766 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9906545877456665 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 & Purity/2FRt*SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.2413194626569748 :"Purity/1FRt0SRqqt0FBt/event_purity0.17885248363018036 & Purity/1FRt1SRqqt1FBt/FBt_purity1 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9936347007751465 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.4888888895511627 particle/accuracy_4_of_51 ) Purity/2FRt3SRqqt1FBt/FRt_purity0.25 ("Purity/*FRt1SRqqt4FBt/event_purity0 7 Purity/2FRt*SRqqt0FBt/FRt_purity0.2093617469072342 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity1 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 :"Purity/*FRt1SRqqt*FBt/event_purity0.10381504893302917 :"Purity/0FRt1SRqqt*FBt/event_purity0.19151823222637177 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity1 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity0.125 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt*SRqqt*FBt/event_purity0 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.23192232847213745 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 ("Purity/3FRt2SRqqt0FBt/event_purity0 ("Purity/*FRt0SRqqt4FBt/event_purity0 ;"Purity/1FRt1SRqqt*FBt/event_purity0.013451669365167618 ("Purity/2FRt2SRqqt1FBt/event_purity0 ("Purity/3FRt*SRqqt0FBt/event_purity0 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 jet/accuracy_4_of_60 :"Purity/1FRt0SRqqt2FBt/event_purity0.07506613433361053 ("Purity/3FRt2SRqqt*FBt/event_purity0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.4000000059604645 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9979513883590698 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9744444489479065 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.3083333373069763 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 ("Purity/0FRt2SRqqt2FBt/event_purity0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ("Purity/3FRt1SRqqt*FBt/event_purity0 +jet/accuracy_3_of_50.06553975492715836 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity0.1875 particle/accuracy_3_of_71 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.9111111164093018 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.22788745164871216 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9873272180557251 ( Purity/2FRt4SRqqt0FBt/FRt_purity0.5 9"Purity/*FRt0SRqqt*FBt/event_purity0.3728640675544739 :"Purity/*FRt2SRqqt1FBt/event_purity0.01221302431076765 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 ("Purity/*FRt2SRqqt2FBt/event_purity0 ("Purity/0FRt3SRqqt1FBt/event_purity0 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9967913627624512 :"Purity/1FRt1SRqqt1FBt/event_purity0.03241582587361336 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.44719386100769043 ("Purity/*FRt4SRqqt1FBt/event_purity0 & Purity/*FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt4SRqqt0FBt/event_purity0 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 <"Purity/2FRt*SRqqt0FBt/event_purity0.0034361679572612047 ("Purity/0FRt3SRqqt*FBt/event_purity0 ("Purity/2FRt*SRqqt3FBt/event_purity0 ("Purity/*FRt3SRqqt2FBt/event_purity0 :"Purity/*FRt0SRqqt2FBt/event_purity0.21443180739879608 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity0.009967588819563389 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.46666666865348816 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity0.5167510509490967 particle/accuracy_5_of_71 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 9"Purity/*FRt0SRqqt3FBt/event_purity0.1011904701590538 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.24998115003108978 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.379761278629303 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.2948717772960663 ) Purity/0FRt*SRqqt4FBt/FBt_purity0.75 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9868420958518982 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2916666567325592 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.4444444477558136 ("Purity/3FRt*SRqqt2FBt/event_purity0 ("Purity/1FRt0SRqqt4FBt/event_purity0 & Purity/2FRt*SRqqt3FBt/FBt_purity1 & Purity/1FRt1SRqqt4FBt/FRt_purity0 8 Purity/*FRt1SRqqt1FBt/FRt_purity0.30190834403038025 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9748391509056091 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 7 Purity/3FRt1SRqqt*FBt/FRt_purity0.2016461193561554 ("Purity/0FRt4SRqqt0FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.45552170276641846 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ("Purity/1FRt0SRqqt3FBt/event_purity0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 9"Purity/0FRt*SRqqt*FBt/event_purity0.3434467613697052 :"Purity/0FRt*SRqqt0FBt/event_purity0.16999465227127075 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ("Purity/0FRt3SRqqt0FBt/event_purity0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.45743611454963684 ;"Purity/2FRt*SRqqt*FBt/event_purity0.005230505019426346 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.2030663937330246 & Purity/3FRt0SRqqt1FBt/FBt_purity1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt3SRqqt*FBt/event_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.20000000298023224 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.21677060425281525 6 Purity/1FRt1SRqqt2FBt/FRt_purity0.434353768825531 *jet/accuracy_1_of_20.6258105635643005 ("Purity/1FRt1SRqqt4FBt/event_purity0 7 Purity/*FRt0SRqqt1FBt/FRt_purity0.3664392828941345 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9876875281333923 8 Purity/*FRt3SRqqt1FBt/FRt_purity0.21666666865348816 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 & Purity/2FRt2SRqqt1FBt/FBt_purity1 & Purity/1FRt0SRqqt4FBt/FRt_purity0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity1 :"Purity/3FRt*SRqqt1FBt/event_purity0.00909090880304575 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.23429951071739197 ("Purity/4FRt1SRqqt0FBt/event_purity0 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.25218409299850464 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity0.25558269023895264 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9834644794464111 9"Purity/0FRt1SRqqt0FBt/event_purity0.2134069800376892 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.2543412744998932 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.42105263471603394 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 :"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.39494043588638306 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.25154465436935425 particle/accuracy_2_of_41 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.1875 & Purity/3FRt2SRqqt*FBt/FBt_purity1 jet/accuracy_2_of_70 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.2505531311035156 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.32568296790122986 *jet/accuracy_2_of_50.3545837104320526 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt3SRqqt1FBt/event_purity0 & Purity/*FRt*SRqqt4FBt/FRt_purity0 & Purity/4FRt*SRqqt0FBt/FRt_purity0 7 Purity/3FRt*SRqqt1FBt/FRt_purity0.3424242436885834 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2513667047023773 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.22613321244716644 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ,particle/sensitivity0.25099170207977295 + Purity/0FRt3SRqqt*FBt/FBt_purity0.8125 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.2380952537059784 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 particle/accuracy_2_of_21 + Purity/0FRt3SRqqt1FBt/FBt_purity0.8125 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.22470787167549133 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 & Purity/3FRt0SRqqt2FBt/FBt_purity1 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8980844020843506 ("Purity/2FRt*SRqqt2FBt/event_purity0 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 particle/accuracy_1_of_21 ("Purity/0FRt0SRqqt4FBt/event_purity0 particle/accuracy_6_of_61 ("Purity/1FRt2SRqqt*FBt/event_purity0 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 ("Purity/3FRt2SRqqt1FBt/event_purity0 :"Purity/*FRt1SRqqt1FBt/event_purity0.08762382715940475 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9993136525154114 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.26251333951950073 ("Purity/*FRt4SRqqt*FBt/event_purity0 & Purity/3FRt2SRqqt1FBt/FBt_purity1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.4740910828113556 :"Purity/0FRt1SRqqt1FBt/event_purity0.14694996178150177 ;"Purity/0FRt2SRqqt*FBt/event_purity0.025064019486308098 ;"Purity/2FRt0SRqqt*FBt/event_purity0.013003498315811157 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 (particle/accuracy0.8394185900688171 jet/accuracy_5_of_70 ("Purity/4FRt*SRqqt0FBt/event_purity0 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9886924624443054 ("Purity/*FRt3SRqqt0FBt/event_purity0 ("Purity/1FRt*SRqqt3FBt/event_purity0 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9285714030265808 & Purity/2FRt2SRqqt*FBt/FBt_purity1 trainer/global_step1811 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 7 Purity/2FRt*SRqqt2FBt/FRt_purity0.4497222304344177 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.21092720329761505 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 *jet/accuracy_1_of_30.7612279057502747 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.427853524684906 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9903225302696228 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.16013072431087494 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ("Purity/1FRt4SRqqt0FBt/event_purity0 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 9"Purity/0FRt*SRqqt1FBt/event_purity0.6333127617835999 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 6 Purity/2FRt0SRqqt0FBt/FRt_purity0.169196218252182 & Purity/3FRt0SRqqt*FBt/FBt_purity1 & Purity/4FRt1SRqqt0FBt/FRt_purity0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 )loss/val_total_loss44.02636196186718 ;"Purity/0FRt2SRqqt1FBt/event_purity0.023476703092455864 *jet/accuracy_1_of_60.9035088419914246 particle/accuracy_2_of_61 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.36640214920043945 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.25755438208580017 :"Purity/1FRt*SRqqt1FBt/event_purity0.13192588090896606 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.37703531980514526 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.2814566493034363 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.8333333730697632 & Purity/2FRt2SRqqt2FBt/FBt_purity1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.2553379535675049 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 ("Purity/3FRt0SRqqt2FBt/event_purity0 6 Purity/*FRt*SRqqt3FBt/FRt_purity0.529411792755127 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9792709946632385 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.41141536831855774 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 & Purity/3FRt1SRqqt*FBt/FBt_purity1 ("Purity/1FRt*SRqqt4FBt/event_purity0 8"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.699999988079071 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt0FBt/FRt_purity0.1555555760860443 ( Purity/*FRt3SRqqt2FBt/FBt_purity0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity0.9889025092124939 :"Purity/*FRt1SRqqt0FBt/event_purity0.11295118182897568 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.25325432419776917 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.2430555373430252 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.22516416013240814 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 9"Purity/0FRt*SRqqt2FBt/event_purity0.2444491684436798 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 &particle/f_score0.327949583530426 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9785056710243225 ( Purity/1FRt0SRqqt3FBt/FRt_purity0.5 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 )jet/accuracy_1_of_40.829477846622467 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.2182447463274002 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.6929824352264404 :"Purity/3FRt0SRqqt*FBt/event_purity0.01515151560306549 ("Purity/0FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt4SRqqt*FBt/event_purity0 6 Purity/0FRt1SRqqt3FBt/FBt_purity0.800000011920929 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity0 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 particle/accuracy_1_of_11 particle/accuracy_5_of_51 & Purity/0FRt2SRqqt2FBt/FBt_purity1 ("Purity/2FRt0SRqqt3FBt/event_purity0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 ("Purity/2FRt1SRqqt0FBt/event_purity0 particle/accuracy_1_of_51 & Purity/3FRt*SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9865448474884033 :"Purity/*FRt0SRqqt0FBt/event_purity0.15 `081650018692017 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2657705843448639 7 Purity/*FRt3SRqqt*FBt/FBt_purity0.9318181872367859 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.23032408952713013 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 particle/accuracy_1_of_31 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.7916666865348816 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.17695888876914978 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.25636059045791626 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.261904776096344 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9761003851890564 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.22578290104866028 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 9"Purity/1FRt*SRqqt2FBt/event_purity0.0475294291973114 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 *jet/accuracy_1_of_10.3589954376220703 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.1111111119389534 jet/accuracy_6_of_70 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.125 7"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.69047611951828 $ _timestamp1.788468294296187e+09 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.23594240844249725 7 Purity/1FRt*SRqqt*FBt/FBt_purity0.9896420240402222 particle/accuracy_2_of_71 & Purity/1FRt0SRqqt4FBt/FBt_purity1 ( Purity/1FRt1SRqqt3FBt/FRt_purity0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt0FBt/event_purity0 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 :"Purity/1FRt*SRqqt*FBt/event_purity0.08922359347343445 8 Purity/1FRt*SRqqt0FBt/FRt_purity0.20853902399539948 9"Purity/1FRt0SRqqt*FBt/event_purity0.1855662316083908 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.4615384638309479 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.2380952537059784 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9738863706588745 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.2187301516532898 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 )jet/accuracy_1_of_50.894727349281311 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 & Purity/3FRt*SRqqt*FBt/FBt_purity1 particle/accuracy_1_of_71 & Purity/1FRt3SRqqt2FBt/FRt_purity1 & Purity/4FRt*SRqqt*FBt/FRt_purity0 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 & Purity/3FRt*SRqqt1FBt/FBt_purity1 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.2142857164144516 ("Purity/3FRt3SRqqt*FBt/event_purity0 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9792627096176147 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ("Purity/4FRt1SRqqt*FBt/event_purity0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ("Purity/4FRt0SRqqt*FBt/event_purity0 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 ("Purity/*FRt1SRqqt3FBt/event_purity0 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9974640011787415 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 ("Purity/*FRt3SRqqt*FBt/event_purity0 9"Purity/0FRt0SRqqt*FBt/event_purity0.8048502206802368 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 ("Purity/1FRt3SRqqt1FBt/event_purity0 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 jet/accuracy_4_of_70 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.2742515504360199 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.21877218782901764 ("Purity/2FRt3SRqqt0FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.21749141812324524 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9988161325454712 7 Purity/*FRt3SRqqt1FBt/FBt_purity0.9523809552192688 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.2874999940395355 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.21051588654518127 :"Purity/*FRt*SRqqt1FBt/event_purity0.39564719796180725 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.2134069800376892 7 Purity/3FRt0SRqqt1FBt/FRt_purity0.5128205418586731 ("Purity/2FRt4SRqqt0FBt/event_purity0 jet/accuracy_6_of_60 & Purity/4FRt1SRqqt*FBt/FRt_purity0 ("Purity/4FRt0SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 8 Purity/2FRt0SRqqt1FBt/FRt_purity0.31179895997047424 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/event_purity0 particle/accuracy_5_of_61 :"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.21153846383094788 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity1 ;"Purity/*FRt2SRqqt*FBt/event_purity0.009965968318283558 7 Purity/3FRt0SRqqt*FBt/FRt_purity0.3181818425655365 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.22807015478610992 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8847736716270447 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914Sai 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.45552170276641846 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ("Purity/1FRt0SRqqt3FBt/event_purity0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 9"Purity/0FRt*SRqqt*FBt/event_purity0.3434467613697052 :"Purity/0FRt*SRqqt0FBt/event_purity0.16999465227127075 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ("Purity/0FRt3SRqqt0FBt/event_purity0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.45743611454963684 ;"Purity/2FRt*SRqqt*FBt/event_purity0.005230505019426346 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.2030663937330246 & Purity/3FRt0SRqqt1FBt/FBt_purity1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt3SRqqt*FBt/event_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.20000000298023224 8 Purity/*FRt2SRqqt0FBt/FRt_purity0.21677060425281525 6 Purity/1FRt1SRqqt2FBt/FRt_purity0.434353768825531 *jet/accuracy_1_of_20.6258105635643005 ("Purity/1FRt1SRqqt4FBt/event_purity0 7 Purity/*FRt0SRqqt1FBt/FRt_purity0.3664392828941345 7 Purity/*FRt2SRqqt*FBt/FBt_purity0.9876875281333923 8 Purity/*FRt3SRqqt1FBt/FRt_purity0.21666666865348816 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 & Purity/2FRt2SRqqt1FBt/FBt_purity1 & Purity/1FRt0SRqqt4FBt/FRt_purity0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity1 :"Purity/3FRt*SRqqt1FBt/event_purity0.00909090880304575 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.23429951071739197 ("Purity/4FRt1SRqqt0FBt/event_purity0 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.25218409299850464 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity0.25558269023895264 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9834644794464111 9"Purity/0FRt1SRqqt0FBt/event_purity0.2134069800376892 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.2543412744998932 8 Purity/2FRt1SRqqt2FBt/FRt_purity0.42105263471603394 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 :"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.39494043588638306 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.25154465436935425 particle/accuracy_2_of_41 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity0.1875 & Purity/3FRt2SRqqt*FBt/FBt_purity1 jet/accuracy_2_of_70 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.2505531311035156 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.32568296790122986 *jet/accuracy_2_of_50.3545837104320526 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt3SRqqt1FBt/event_purity0 & Purity/*FRt*SRqqt4FBt/FRt_purity0 & Purity/4FRt*SRqqt0FBt/FRt_purity0 7 Purity/3FRt*SRqqt1FBt/FRt_purity0.3424242436885834 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2513667047023773 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.22613321244716644 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ,particle/sensitivity0.25099170207977295 + Purity/0FRt3SRqqt*FBt/FBt_purity0.8125 7 Purity/3FRt3SRqqt*FBt/FRt_purity0.2380952537059784 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 particle/accuracy_2_of_21 + Purity/0FRt3SRqqt1FBt/FBt_purity0.8125 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.22470787167549133 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 & Purity/3FRt0SRqqt2FBt/FBt_purity1 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8980844020843506 ("Purity/2FRt*SRqqt2FBt/event_purity0 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 particle/accuracy_1_of_21 ("Purity/0FRt0SRqqt4FBt/event_purity0 particle/accuracy_6_of_61 ("Purity/1FRt2SRqqt*FBt/event_purity0 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 ("Purity/3FRt2SRqqt1FBt/event_purity0 :"Purity/*FRt1SRqqt1FBt/event_purity0.08762382715940475 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9993136525154114 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.26251333951950073 ("Purity/*FRt4SRqqt*FBt/event_purity0 & Purity/3FRt2SRqqt1FBt/FBt_purity1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.4740910828113556 :"Purity/0FRt1SRqqt1FBt/event_purity0.14694996178150177 ;"Purity/0FRt2SRqqt*FBt/event_purity0.025064019486308098 ;"Purity/2FRt0SRqqt*FBt/event_purity0.013003498315811157 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 (particle/accuracy0.8394185900688171 jet/accuracy_5_of_70 ("Purity/4FRt*SRqqt0FBt/event_purity0 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9886924624443054 ("Purity/*FRt3SRqqt0FBt/event_purity0 ("Purity/1FRt*SRqqt3FBt/event_purity0 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9285714030265808 & Purity/2FRt2SRqqt*FBt/FBt_purity1 trainer/global_step1811 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 7 Purity/2FRt*SRqqt2FBt/FRt_purity0.4497222304344177 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.21092720329761505 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 *jet/accuracy_1_of_30.7612279057502747 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.427853524684906 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9903225302696228 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.16013072431087494 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ("Purity/1FRt4SRqqt0FBt/event_purity0 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 9"Purity/0FRt*SRqqt1FBt/event_purity0.6333127617835999 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 6 Purity/2FRt0SRqqt0FBt/FRt_purity0.169196218252182 & Purity/3FRt0SRqqt*FBt/FBt_purity1 & Purity/4FRt1SRqqt0FBt/FRt_purity0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 )loss/val_total_loss44.02636196186718 ;"Purity/0FRt2SRqqt1FBt/event_purity0.023476703092455864 *jet/accuracy_1_of_60.9035088419914246 particle/accuracy_2_of_61 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.36640214920043945 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.25755438208580017 :"Purity/1FRt*SRqqt1FBt/event_purity0.13192588090896606 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.37703531980514526 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.2814566493034363 ("Purity/3FRt1SRqqt2FBt/event_purity0 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.8333333730697632 & Purity/2FRt2SRqqt2FBt/FBt_purity1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.2553379535675049 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 ("Purity/3FRt0SRqqt2FBt/event_purity0 6 Purity/*FRt*SRqqt3FBt/FRt_purity0.529411792755127 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9792709946632385 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.41141536831855774 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 & Purity/3FRt1SRqqt*FBt/FBt_purity1 ("Purity/1FRt*SRqqt4FBt/event_purity0 8"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.699999988079071 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt0FBt/FRt_purity0.1555555760860443 ( Purity/*FRt3SRqqt2FBt/FBt_purity0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity0.9889025092124939 :"Purity/*FRt1SRqqt0FBt/event_purity0.11295118182897568 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.25325432419776917 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.2430555373430252 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.22516416013240814 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 9"Purity/0FRt*SRqqt2FBt/event_purity0.2444491684436798 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 &particle/f_score0.327949583530426 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9785056710243225 ( Purity/1FRt0SRqqt3FBt/FRt_purity0.5 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 )jet/accuracy_1_of_40.829477846622467 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.2182447463274002 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.6929824352264404 :"Purity/3FRt0SRqqt*FBt/event_purity0.01515151560306549 ("Purity/0FRt*SRqqt4FBt/event_purity0 ("Purity/2FRt4SRqqt*FBt/event_purity0 6 Purity/0FRt1SRqqt3FBt/FBt_purity0.800000011920929 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity0 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 particle/accuracy_1_of_11 particle/accuracy_5_of_51 & Purity/0FRt2SRqqt2FBt/FBt_purity1 ("Purity/2FRt0SRqqt3FBt/event_purity0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 ("Purity/2FRt1SRqqt0FBt/event_purity0 particle/accuracy_1_of_51 & Purity/3FRt*SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9865448474884033 :"Purity/*FRt0SRqqt0FBt/event_purity0.15081650018692017 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2657705843448639 7 Purity/*FRt3SRqqt*FBt/FBt_purity0.9318181872367859 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.23032408952713013 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 particle/accuracy_1_of_31 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.7916666865348816 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.17695888876914978 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.25636059045791626 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 & Purity/3FRt1SRqqt2FBt/FBt_purity1 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.261904776096344 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9761003851890564 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.22578290104866028 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 9"Purity/1FRt*SRqqt2FBt/event_purity0.0475294291973114 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 *jet/accuracy_1_of_10.3589954376220703 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.1111111119389534 jet/accuracy_6_of_70 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.125 7"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.69047611951828 $ _timestamp1.788468294296187e+09 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.23594240844249725 7 Purity/1FRt*SRqqt*FBt/FBt_purity0.9896420240402222 particle/accuracy_2_of_71 & Purity/1FRt0SRqqt4FBt/FBt_purity1 ( Purity/1FRt1SRqqt3FBt/FRt_purity0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt0FBt/event_purity0 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 :"Purity/1FRt*SRqqt*FBt/event_purity0.08922359347343445 8 Purity/1FRt*SRqqt0FBt/FRt_purity0.20853902399539948 9"Purity/1FRt0SRqqt*FBt/event_purity0.1855662316083908 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 9"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.4615384638309479 7 Purity/3FRt3SRqqt0FBt/FRt_purity0.2380952537059784 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9738863706588745 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.2187301516532898 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 )jet/accuracy_1_of_50.894727349281311 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 & Purity/3FRt*SRqqt*FBt/FBt_purity1 particle/accuracy_1_of_71 & Purity/1FRt3SRqqt2FBt/FRt_purity1 & Purity/4FRt*SRqqt*FBt/FRt_purity0 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 & Purity/3FRt*SRqqt1FBt/FBt_purity1 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.2142857164144516 ("Purity/3FRt3SRqqt*FBt/event_purity0 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9792627096176147 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 ("Purity/4FRt1SRqqt*FBt/event_purity0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity1 ("Purity/4FRt0SRqqt*FBt/event_purity0 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 ("Purity/*FRt1SRqqt3FBt/event_purity0 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9974640011787415 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 ("Purity/*FRt3SRqqt*FBt/event_purity0 9"Purity/0FRt0SRqqt*FBt/event_purity0.8048502206802368 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 ("Purity/1FRt3SRqqt1FBt/event_purity0 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 jet/accuracy_4_of_70 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.2742515504360199 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.21877218782901764 ("Purity/2FRt3SRqqt0FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.21749141812324524 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9988161325454712 7 Purity/*FRt3SRqqt1FBt/FBt_purity0.9523809552192688 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.2874999940395355 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.21051588654518127 :"Purity/*FRt*SRqqt1FBt/event_purity0.39564719796180725 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.2134069800376892 7 Purity/3FRt0SRqqt1FBt/FRt_purity0.5128205418586731 ("Purity/2FRt4SRqqt0FBt/event_purity0 jet/accuracy_6_of_60 & Purity/4FRt1SRqqt*FBt/FRt_purity0 ("Purity/4FRt0SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 8 Purity/2FRt0SRqqt1FBt/FRt_purity0.31179895997047424 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/event_purity0 particle/accuracy_5_of_61 :"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.21153846383094788 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity1 ;"Purity/*FRt2SRqqt*FBt/event_purity0.009965968318283558 7 Purity/3FRt0SRqqt*FBt/FRt_purity0.3181818425655365 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.22807015478610992 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8847736716270447 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.22889353334903717 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 +particle/specificity0.9483219385147095 :"Purity/0FRt0SRqqt3FBt/event_purity0.13333332538604736 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 ("Purity/1FRt2SRqqt0FBt/event_purity0 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 particle/accuracy_1_of_41 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity0 7 Purity/*FRt1SRqqt1FBt/FBt_purity0.9969988465309143 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 ("Purity/1FRt1SRqqt2FBt/event_purity0 ("Purity/2FRt1SRqqt1FBt/event_purity0 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 particle/accuracy_2_of_31 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.20000000298023224 ;"Purity/2FRt*SRqqt1FBt/event_purity0.014498612843453884 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 & Purity/3FRt1SRqqt1FBt/FBt_purity1 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.24814090132713318 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 ) Purity/*FRt4SRqqt*FBt/FRt_purity0.25 ) Purity/0FRt0SRqqt4FBt/FBt_purity0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity0 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.3060974180698395 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 ("Purity/4FRt2SRqqt*FBt/event_purity0 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.32670044898986816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.4103369414806366 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9758555293083191 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 & Purity/1FRt*SRqqt4FBt/FRt_purity0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.2670695185661316 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.3333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.25 :"Purity/*FRt*SRqqt3FBt/event_purity0.09195402264595032 8 Purity/1FRt0SRqqt*FBt/FRt_purity0.25639331340789795 8 Purity/1FRt2SRqqt1FBt/FRt_purity0.25460830330848694 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.38945698738098145 ("Purity/2FRt2SRqqt*FBt/event_purity0 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.4041556417942047 ("Purity/0FRt0SRqqt0FBt/event_purity1 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.25935959815979004 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.22769273817539215 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 +jet/accuracy_3_of_30.01847144216299057 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.36626383662223816 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.22991222143173218 ("Purity/2FRt2SRqqt0FBt/event_purity0 ("Purity/3FRt0SRqqt0FBt/event_purity0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.21195651590824127 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 particle/accuracy_3_of_41 jet/accuracy_5_of_60 8"Purity/1FRt0SRqqt1FBt/event_purity0.239681676030159 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.408525288105011 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 particle/accuracy_4_of_71 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 & Purity/*FRt0SRqqt4FBt/FRt_purity0 & Purity/1FRt4SRqqt0FBt/FRt_purity0 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 9"Purity/0FRt0SRqqt2FBt/event_purity0.3033808469772339 , Purity/1FRt3SRqqt*FBt/FBt_purity0.96875 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.2217460423707962 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 particle/accuracy_3_of_51 & Purity/0FRt4SRqqt*FBt/FBt_purity1 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 9"Purity/0FRt0SRqqt1FBt/event_purity0.9993136525154114 :"Purity/0FRt2SRqqt0FBt/event_purity0.02492099069058895 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.25453150272369385 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.125 6 Purity/0FRt*SRqqt*FBt/FBt_purity0.983864426612854 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 particle/accuracy_3_of_31 & Purity/4FRt0SRqqt*FBt/FRt_purity0 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9945845007896423 8 Purity/1FRt*SRqqt2FBt/FRt_purity0.49648764729499817 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.22720058262348175 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 8 Purity/1FRt1SRqqt*FBt/FRt_purity0.25244107842445374 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 & Purity/*FRt1SRqqt4FBt/FRt_purity0 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 7 Purity/2FRt1SRqqt1FBt/FRt_purity0.3188747763633728 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 particle/accuracy_6_of_71 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 :"Purity/0FRt*SRqqt3FBt/event_purity0.12345678359270096 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 8 Purity/1FRt1SRqqt1FBt/FRt_purity0.29422077536582947 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.4037037491798401 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.24702021479606628 7 Purity/*FRt0SRqqt2FBt/FRt_purity0.4928172826766968 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 7 Purity/2FRt1SRqqt*FBt/FRt_purity0.2510954737663269 8 Purity/2FRt3SRqqt0FBt/FRt_purity0.31617963314056396 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 particle/accuracy_4_of_41 & Purity/2FRt3SRqqt*FBt/FBt_purity1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity0.125 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 ("Purity/2FRt3SRqqt*FBt/event_purity0 *jet/accuracy_2_of_40.2889387011528015 +jet/accuracy_3_of_40.04575696960091591 ("Purity/1FRt4SRqqt*FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FRt_purity1 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 8 Purity/2FRt*SRqqt1FBt/FRt_purity0.32143744826316833 ,jet/accuracy_4_of_50.007857734337449074 & Purity/2FRt3SRqqt1FBt/FBt_purity1 _runtime3184.338633593 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 +jet/accuracy_2_of_30.19193962216377258 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.2300773411989212 8 Purity/2FRt*SRqqt*FBt/FRt_purity0.24266502261161804 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.6754385828971863 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8492063879966736 ;"Purity/*FRt1SRqqt2FBt/event_purity0.033591676503419876 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9867276549339294 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9958243370056152 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 particle/accuracy_2_of_51 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.261904776096344 _step146 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.21345369517803192 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ("Purity/3FRt1SRqqt1FBt/event_purity0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 & Purity/0FRt4SRqqt1FBt/FBt_purity1 9"Purity/*FRt0SRqqt1FBt/event_purity0.6746923327445984 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 ( Purity/2FRt4SRqqt*FBt/FRt_purity0.5 ("Purity/0FRt1SRqqt3FBt/event_purity0 :"Purity/*FRt*SRqqt2FBt/event_purity0.16256065666675568 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.9113095998764038 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.2278536856174469 ("Purity/2FRt1SRqqt*FBt/event_purity0 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity 0.1953125 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 +jet/accuracy_2_of_60.28947368264198303 ("Purity/1FRt1SRqqt3FBt/event_purity0 9"Purity/*FRt*SRqqt*FBt/event_purity0.1882375031709671 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 particle/accuracy_1_of_61 jet/accuracy_1_of_71 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity0.9723148345947266 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.27283933758735657 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 -jet/accuracy_4_of_40.0010080644860863686 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 7 Purity/1FRt*SRqqt3FBt/FBt_purity0.9270833134651184 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 :"Purity/3FRt*SRqqt*FBt/event_purity0.00358422938734293 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt0SRqqt2FBt/event_purity0 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.22511614859104156 @&Purity/1FRt0SRqqt3FBt/event_proportion0.00107106857467 )44275 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.25 ("Purity/*FRt2SRqqt3FBt/event_purity0 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 ;"Purity/1FRt1SRqqt0FBt/event_purity0.008387289941310883 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 :"Purity/0FRt1SRqqt2FBt/event_purity0.06296297162771225 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.31481480598449707 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 jet/accuracy_5_of_50 7validation_average_jet_accuracy0.34588029980659485 ("Purity/0FRt4SRqqt*FBt/event_purity0 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.3333333432674408 5 Purity/*FRt*SRqqt0FBt/FRt_purity0.20804363489151 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.38657405972480774 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ) Purity/*FRt4SRqqt0FBt/FRt_purity0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity0 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9868655800819397 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.39307278394699097 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 6 Purity/*FRt0SRqqt4FBt/FBt_purity0.800000011920929 & Purity/*FRt4SRqqt*FBt/FBt_purity1 ("Purity/1FRt2SRqqt2FBt/event_purity0 epoch2 7 Purity/1FRt2SRqqt0FBt/FRt_purity0.2162618190050125 ("Purity/1FRt3SRqqt0FBt/event_purity0 ("Purity/2FRt1SRqqt2FBt/event_purity0 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 :"Purity/3FRt0SRqqt1FBt/event_purity0.03846153989434242 & Purity/1FRt4SRqqt*FBt/FRt_purity0 ("Purity/1FRt3SRqqt2FBt/event_purity0 ("Purity/2FRt2SRqqt2FBt/event_purity0 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9860703945159912 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.21237719058990479 ("Purity/3FRt3SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.3924858868122101 8 Purity/1FRt0SRqqt0FBt/FRt_purity0.17885248363018036 ("Purity/*FRt3SRqqt1FBt/event_purity0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.4327944815158844 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9907825589179993 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9761905074119568 ("Purity/1FRt2SRqqt1FBt/event_purity0 jet/accuracy_3_of_70 jet/accuracy_7_of_70 * Purity/1FRt*SRqqt4FBt/FBt_purity0.875 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 & Purity/4FRt0SRqqt0FBt/FRt_purity0 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.20000000298023224 +jet/accuracy_2_of_20.08624794334173203 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.3333333432674408 particle/accuracy_4_of_61 & Purity/2FRt0SRqqt3FBt/FBt_purity1 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.2811186909675598 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.2664150893688202 :"Purity/1FRt*SRqqt0FBt/event_purity0.07845912128686905 ("Purity/*FRt4SRqqt0FBt/event_purity0 6 Purity/*FRt*SRqqt2FBt/FBt_purity0.978058397769928 ( Purity/*FRt0SRqqt3FBt/FRt_purity0.5 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.3935227692127228 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt3SRqqt*FBt/FRt_purity0.2473374307155609 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/3FRt4SRqqt*FBt/event_purity0 :"Purity/*FRt*SRqqt0FBt/event_purity0.10097857564687729 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 ;"Purity/2FRt0SRqqt0FBt/event_purity0.010075408965349197 ;"Purity/2FRt0SRqqt1FBt/event_purity0.027037037536501884 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 8 Purity/2FRt1SRqqt0FBt/FRt_purity0.22262871265411377 7 Purity/2FRt3SRqqt*FBt/FRt_purity0.3014761805534363 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 +jet/accuracy_3_of_60.05263157933950424 particle/accuracy_3_of_61 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.43217846751213074 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.2937720715999603 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.23262843489646912 8 Purity/*FRt2SRqqt2FBt/FRt_purity0.44871795177459717 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9791666865348816 particle/accuracy_7_of_71 ("Purity/0FRt2SRqqt3FBt/event_purity0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.41228678822517395 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 , Purity/1FRt*SRqqt3FBt/FRt_purity0.53125 7 Purity/1FRt0SRqqt2FBt/FBt_purity0.9849491119384766 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9906545877456665 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 & Purity/2FRt*SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.2413194626569748 :"Purity/1FRt0SRqqt0FBt/event_purity0.17885248363018036 & Purity/1FRt1SRqqt1FBt/FBt_purity1 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9936347007751465 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.4888888895511627 particle/accuracy_4_of_51 ) Purity/2FRt3SRqqt1FBt/FRt_purity0.25 ("Purity/*FRt1SRqqt4FBt/event_purity0 7 Purity/2FRt*SRqqt0FBt/FRt_purity0.2093617469072342 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity1 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 :"Purity/*FRt1SRqqt*FBt/event_purity0.10381504893302917 :"Purity/0FRt1SRqqt*FBt/event_purity0.19151823222637177 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity1 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity0.125 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt*SRqqt*FBt/event_purity0 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.23192232847213745 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 ("Purity/3FRt2SRqqt0FBt/event_purity0 ("Purity/*FRt0SRqqt4FBt/event_purity0 ;"Purity/1FRt1SRqqt*FBt/event_purity0.013451669365167618 ("Purity/2FRt2SRqqt1FBt/event_purity0 ("Purity/3FRt*SRqqt0FBt/event_purity0 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 jet/accuracy_4_of_60 :"Purity/1FRt0SRqqt2FBt/event_purity0.07506613433361053 ("Purity/3FRt2SRqqt*FBt/event_purity0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.4000000059604645 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9979513883590698 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9744444489479065 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.3083333373069763 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 ("Purity/0FRt2SRqqt2FBt/event_purity0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 ("Purity/3FRt1SRqqt*FBt/event_purity0 +jet/accuracy_3_of_50.06553975492715836 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity0.1875 particle/accuracy_3_of_71 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.9111111164093018 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.22788745164871216 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9873272180557251 ( Purity/2FRt4SRqqt0FBt/FRt_purity0.5 9"Purity/*FRt0SRqqt*FBt/event_purity0.3728640675544739 :"Purity/*FRt2SRqqt1FBt/event_purity0.01221302431076765 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 ("Purity/*FRt2SRqqt2FBt/event_purity0 ("Purity/0FRt3SRqqt1FBt/event_purity0 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9967913627624512 :"Purity/1FRt1SRqqt1FBt/event_purity0.03241582587361336 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.44719386100769043 ("Purity/*FRt4SRqqt1FBt/event_purity0 & Purity/*FRt4SRqqt1FBt/FBt_purity1 ("Purity/3FRt4SRqqt0FBt/event_purity0 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 <"Purity/2FRt*SRqqt0FBt/event_purity0.0034361679572612047 ("Purity/0FRt3SRqqt*FBt/event_purity0 ("Purity/2FRt*SRqqt3FBt/event_purity0 ("Purity/*FRt3SRqqt2FBt/event_purity0 :"Purity/*FRt0SRqqt2FBt/event_purity0.21443180739879608 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity0.009967588819563389 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.46666666865348816 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity0.5167510509490967 particle/accuracy_5_of_71 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 9"Purity/*FRt0SRqqt3FBt/event_purity0.1011904701590538 8 Purity/*FRt1SRqqt*FBt/FRt_purity0.24998115003108978 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.379761278629303 7 Purity/3FRt1SRqqt1FBt/FRt_purity0.2948717772960663 ) Purity/0FRt*SRqqt4FBt/FBt_purity0.75 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9868420958518982 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2916666567325592 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.4444444477558136 ("Purity/3FRt*SRqqt2FBt/event_purity0 ("Purity/1FRt0SRqqt4FBt/event_purity0 & Purity/2FRt*SRqqt3FBt/FBt_purity1 & Purity/1FRt1SRqqt4FBt/FRt_purity0 8 Purity/*FRt1SRqqt1FBt/FRt_purity0.30190834403038025 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9748391509056091 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 7 Purity/3FRt1SRqqt*FBt/FRt_purity0.2016461193561554 ("Purity/0FRt4SRqqt0FBt/event_purity0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity1oz(: и gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp33.0 gpu.0.powerWatts92.612' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent18.5224 gpu.0.smClock1275 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0"q(: иڲ# disk.dm-0.out73895.61669921875 disk.sda1.in0 cpu25.23245422948744 disk.sda.out 97.9609375 disk.sda2.in0 disk.sda2.out0# disk.nvme0c0n1.in 7319.98828125( disk.nvme0c0n1.out73904.74169921875 disk.dm-1.out 97.9609375 disk.drbd1260.out0.4375 disk.dm-0.in 7319.98828125) disk./.usagePercent77.25803022928773$ disk./.usageGB1151.200481414795! disk.nvme0n1.in 7319.98828125 disk.sda3.in0 disk.sda3.out 97.9609375$ memory_percent85.92071533203125" proc.memory.rssMB 4403.2421875) proc.memory.percent26.87525749206543 disk.drbd1260.in0& disk.nvme0n1.out73904.86669921875 disk.sda1.out0# proc.memory.availableMB2306.75 proc.cpu.threads17 network.sent 4.051082e+06 network.recv 1.004077e+06 disk.sda.in0 disk.dm-1.in0(: ߸ gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts249.227' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.8454 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ETu(: ߸֝% disk./.usageGB1151.3211975097656& disk.nvme0n1.out75413.15185546875 disk.sda3.out 98.578125 disk.sda.in0# disk.nvme0c0n1.in 7335.57421875( disk.nvme0c0n1.out75413.15185546875! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 network.sent 4.153773e+06! disk.nvme0n1.in 7335.57421875# disk.dm-0.out75413.15185546875 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.out 98.578125 disk.drbd1260.in0 disk.sda.out 98.578125) disk./.usagePercent77.26613158770876 disk.dm-0.in 7335.57421875 disk.sda1.in0 disk.sda1.out0 disk.drbd1260.out0.4375# memory_percent77.8507947921753) proc.memory.availableMB 3628.92578125 network.recv 1.010374e+06 disk.dm-1.in0 cpu25.233522006667506 proc.cpu.threads17ſ (:  gpu.0.gpu85.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp44.0 gpu.0.powerWatts257.067' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent51.413399999999996 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0/my(: ӹ disk.sda1.out0 disk.sda3.out 99.01953125 disk.sda.out 99.01953125 disk.sda2.out0 disk.dm-1.in0 disk.drbd1260.in0! disk.nvme0n1.in 7335.57421875 network.sent 4.159715e+06) disk./.usagePercent77.27452197863595! disk.dm-0.out75688.556640625 disk.sda.in0 disk.sda2.in0 disk.drbd1260.out 0.44140625$ memory_percent77.84931659698486) proc.memory.availableMB 3629.16796875% disk./.usageGB1151.4462203979492 disk.dm-0.in 7335.57421875 disk.sda1.in0# disk.nvme0c0n1.in 7335.57421875& disk.nvme0c0n1.out75688.556640625! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 disk.sda3.in0 disk.dm-1.out 99.01953125$ disk.nvme0n1.out75688.556640625 cpu25.23428987419031 proc.cpu.threads17 network.recv 1.012956e+06 1,(: į gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts236.697' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.3394 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors03+k(:  disk.dm-1.in0) disk./.usagePercent77.30814447213787! disk.dm-0.out76047.861328125 disk.dm-1.out99.5625 disk.drbd1260.in0 disk.drbd1260.out 0.44140625* proc.memory.percent19.058895111083984 disk.sda.out99.5625$ disk.nvme0n1.out76047.861328125 disk.dm-0.in 7335.57421875 disk.sda3.in0 disk.sda3.out99.5625 disk.sda2.out0# disk.nvme0c0n1.in 7335.57421875' proc.memory.availableMB 3629.421875 network.sent 4.163988e+06% disk./.usageGB1151.9472198486328& disk.nvme0c0n1.out76047.861328125# memory_percent77.8477668762207! proc.memory.rssMB 3122.609375 cpu25.235194397643973 proc.cpu.threads17 disk.sda.in0 network.recv 1.017656e+06! disk.nvme0n1.in 7335.57421875 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0)ŝ(: ߻ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts230.023' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.0046 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0U'z(:  cpu25.236120908190834% disk.nvme0n1.out76550.0419921875 network.sent 4.169552e+06) disk./.usagePercent77.30807765417143% disk./.usageGB1151.9462242126465 disk.sda2.out0# disk.nvme0c0n1.in 7335.57421875 disk.dm-1.in0 disk.dm-0.in 7335.57421875 network.recv 1.021812e+06 disk.drbd1260.in0" disk.dm-0.out76550.0419921875 disk.sda3.in0 disk.sda3.out 99.98828125' proc.memory.availableMB 3629.171875 proc.cpu.threads17! disk.nvme0n1.in 7335.57421875 disk.sda2.in0 disk.drbd1260.out 0.4453125 disk.sda1.out0 disk.sda.in0 disk.sda.out 99.98828125$ memory_percent77.84929275512695! proc.memory.rssMB 3122.609375' disk.nvme0c0n1.out76550.0419921875 disk.dm-1.out 99.98828125 disk.sda1.in0* proc.memory.percent19.058895111083984eR2- _wandbruntime3257  _runtime3257 & lr-AdamW/pg10.0010317861904610543 & lr-AdamW/pg20.0010317861904610543 trainer/global_step1849 % _timestamp1.7884683670753977e+09 _runtime3257.116345605 _step147qE & lr-AdamW/pg10.0010317861904610543 & lr-AdamW/pg20.0010317861904610543 trainer/global_step1849 % _timestamp1.7884683670753977e+09 _runtime3257.116345605 _step147e2- _wandbruntime3257  _runtime3257C'O ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4290143974532 $ _timestamp1.788468367079905e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed82111099317760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 trainer/global_step1849 _runtime 3257.12150395 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated403772191398912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed21420048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed87600896501760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated391621467984336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed8211194 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated61496543 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed34802365 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed387328784213424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated408070496353280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed408067954562048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _step148 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5489856123392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5489797184000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed26674488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed34802365 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4295788591104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated26674494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated387328809641392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated21420497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed61476853 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed403772165970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated34822049 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed408067954562048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated82111128415744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated87600984539136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4295788591104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated26674494 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8211197 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated13209300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4292658342944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated34822049 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed408067954562048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 _step148 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5489856123392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5489797184000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed26674488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed34802365 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4295788591104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated26674494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated387328809641392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak876 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated21420497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed61476853 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed403772165970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated34822049 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed408067954562048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 NFDeviceStatsZVMonitor.on_train_batch_start/inactive_split.large_pool.peak328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated82111128415744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated87600984539136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4295788591104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated26674494 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8211197 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated13209300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4292658342944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated34822049 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed21431293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13215767 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 % _timestamp1.7884683685620737e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated403983181774336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4300580184064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4300580184064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated387531338350976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated61528801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated26688375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed26688369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated61528801 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated13216224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5492781297152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated8215529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated34840426&'M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4294930598006 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed21431293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13215767 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 % _timestamp1.7884683685620737e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated403983181774336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4300580184064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4300580184064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated387531338350976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated61528801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated26688375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed26688369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated61528801 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated13216224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5492781297152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated8215529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated34840426 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated27368544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 trainer/global_step1899 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated27368544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated35742951 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed35723267 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated418733803771904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed63091805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated414321710170112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated63111495 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed27368538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5635382790144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4406298257652 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed414321684742144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed13553374 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8427797 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated13553829 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated35742951 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated84314196882944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5635321753600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed27368538 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated418733803771904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed418731261980672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated27368544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 trainer/global_step1899 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated27368544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated35742951 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed35723267 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated418733803771904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed63091805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated414321710170112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated63111495 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed27368538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5635382790144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4406298257652 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed414321684742144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed13553374 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8427797 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated13553829 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated35742951 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated84314196882944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5635321753600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed27368538 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated418733803771904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed418731261980672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated35761328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed63124037 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5638285444096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 $ _timestamp1.788468452781227e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed27382419 _step153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated414532700545536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated63143753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated414532700545536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed21992345 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13560219 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed35741618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed397657748402208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4414368831488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4406056144094 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed414532675117568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated27382425 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5638224420864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 trainer/global_step1899 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed8432126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated402066344342890 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed63124037 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed418944527572480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated63143753 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed27382419 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated89996543696384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed89996453575168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed84358229154304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61023232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed402063804546302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed414532675117568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated8432129 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated35761328 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed63124037 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 5638285444096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 $ _timestamp1.788468452781227e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed27382419 _step153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated414532700545536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated63143753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated414532700545536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed21992345 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13560219 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed35741618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed397657748402208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4414368831488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4406056144094 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed414532675117568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated27382425 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5638224420864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 trainer/global_step1899 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed8432126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated402066344342890 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed63124037 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed418944527572480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated63143753 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed27382419 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated89996543696384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed89996453575168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed84358229154304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61023232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed402063804546302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed414532675117568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated8432129 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed429394569399296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed424871203513344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5780886971392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4519938172360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4517423803948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed28062588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4523365885952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated424871228941312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated64726447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed13897510 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated28062594 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated412101618772152 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated86517265350144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed8644394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0)O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed429394569399296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8644397 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed86517236252160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 _runtime3424.513672643 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed412099078975772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4525882249216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 % _timestamp1.7884685344726558e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed4075 $81655171824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated424871228941312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed92298064284160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated13897974 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed64706757 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4523365885952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed28062588 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed36644169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed36644169 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated407581680599792 _step156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated36663853 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated22542371 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated36663853 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4525882249216 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated86517265350144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed8644394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed429394569399296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed424871203513344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 5780886971392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4519938172360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4517423803948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1197 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed28062588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4523365885952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated424871228941312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated64726447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed13897510 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated28062594 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated412101618772152 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4522210427422 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed425082193888768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed86561297621504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated64758705 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated407784209309376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated429610376795648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated425082219316736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed429607834991104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated13904771 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated92345115921920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated28076475 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed28076469 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed8648726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13904304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed92345027897856 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5783730276352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated86561326719488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed64738989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 $ _timestamp1.788468535960526e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated412306419736798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed28076469 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed36662520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed64738989 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed22553030 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed407784183881408 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4525641102336 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5783730276352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated86561326719488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed64738989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 $ _timestamp1.788468535960526e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated412306419736798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed28076469 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed36662520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed64738989 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed22553030 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed407784183881408 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4525641102336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4522210427422 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed425082193888768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed86561297621504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated64758705 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated407784209309376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated429610376795648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated425082219316736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed429607834991104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated13904771 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated92345115921920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated28076475 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed28076469 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed8648726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed13904304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed92345027897856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated23103861 trainer/global_step1999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed440057876817920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated440060418609152 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated417708116078992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed94646739234816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated440060418609152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated435420747712512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed14242400 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated94646827272192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated435420747712512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed28756638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed37565071 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4637154533376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8860997 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated422341694166060 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5926434515456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed422339154369680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated23103861 trainer/global_step1999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed440057876817920 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated440060418609152 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated417708116078992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed94646739234816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated440060418609152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated435420747712512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed14242400 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated94646827272192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated435420747712512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed28756638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed37565071 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4637154533376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated8860997 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated422341694166060 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 5926434515456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed422339154369680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated66373657 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed66353941 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step1999 _step161 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated37603132 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4639429749760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed23114596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated417910644788576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 % _timestamp1.7884686190090134e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freedYJ<0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4635850342130 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated28770525 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated28770525 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated14249723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated88764395186688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5929338568192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated422546495130706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated435631738087936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4641946126336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed435631712659968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated23115052 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed88764366088704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4633335973510 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated66373657 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed28770519 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed28770519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed66353941 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated435631738087936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4639429749760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed37583422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576RU'M I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 % _timestamp1.7884686190090134e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4635850342130 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated28770525 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated28770525 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated14249723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated88764395186688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 5929338568192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated422546495130706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated435631738087936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4641946126336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed435631712659968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated23115052 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed88764366088704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4633335973510 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated66373657 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed28770519 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed28770519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed66353941 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated435631738087936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4639429749760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed37583422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed417910619360608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated66373657 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed66353941 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step1999 _step161 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated37603132 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4639429749760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed23114596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated417910644788576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0У2- _wandbruntime3509  _runtime3509ݕ )Z /loss/FRt2/assignment_loss4.133792438368866 2loss/SRqqt3/assignment_loss1.7751374752193256 1loss/SRqqt4/detection_loss1.6177330495073245 1loss/FBt1/assignment_loss0.16058599505584253 1loss/SRqqt3/detection_loss2.4409018711767336 % _timestamp1.7884686190108926e+09 _step162 1loss/SRqqt2/assignment_loss1.781267024910272 1loss/SRqqt2/detection_loss2.5478435309965217 trainer/global_step1999 2loss/SRqqt4/assignment_loss1.5782514891945398 /loss/FBt1/detection_loss0.7326537469236104 /loss/FBt3/detection_loss2.7243500842767605 .loss/FBt4/detection_loss2.905680913191575 _runtime3509.051278434 0loss/FBt4/assignment_loss0.2923740633257414 epoch3 0loss/FRt4/assignment_loss3.0232615407842856 2loss/SRqqt1/assignment_loss0.9000364807857708 /loss/FRt2/detection_loss2.6282894352208013 .loss/FRt3/detection_loss2.666347862025838 .loss/FBt2/detection_loss3.619507751681588 0loss/FRt1/assignment_loss1.8048070970545993 0loss/FRt3/assignment_loss3.3691425419148104 1loss/FBt3/assignment_loss0.11433958104702041 /loss/FRt1/detection_loss1.9401926530546016 -loss/FRt4/detection_loss2.22445375400667 1loss/SRqqt1/detection_loss1.1286167891557553 /loss/FBt2/assignment_loss0.161796693129211 &loss/total_loss46.271363862008066V_ /loss/FBt2/assignment_loss0.161796693129211 &loss/total_loss46.271363862008066 /loss/FRt2/assignment_loss4.133792438368866 2loss/SRqqt3/assignment_loss1.7751374752193256 1loss/SRqqt4/detection_loss1.6177330495073245 1loss/FBt1/assignment_loss0.16058599505584253 1loss/SRqqt3/detection_loss2.4409018711767336 % _timestamp1.7884686190108926e+09 _step162 1loss/SRqqt2/assignment_loss1.781267024910272 1loss/SRqqt2/detection_loss2.5478435309965217 trainer/global_step1999 2loss/SRqqt4/assignment_loss1.5782514891945398 /loss/FBt1/detection_loss0.7326537469236104 /loss/FBt3/detection_loss2.7243500842767605 .loss/FBt4/detection_loss2.905680913191575 _runtime3509.051278434 0loss/FBt4/assignment_loss0.2923740633257414 epoch3 0loss/FRt4/assignment_loss3.0232615407842856 2loss/SRqqt1/assignment_loss0.9000364807857708 /loss/FRt2/detection_loss2.6282894352208013 .loss/FRt3/detection_loss2.666347862025838 .loss/FBt2/detection_loss3.619507751681588 0loss/FRt1/assignment_loss1.8048070970545993 0loss/FRt3/assignment_loss3.3691425419148104 1loss/FBt3/assignment_loss0.11433958104702041 /loss/FRt1/detection_loss1.9401926530546016 -loss/FRt4/detection_loss2.22445375400667 1loss/SRqqt1/detection_loss1.1286167891557553PA$5(: ƶ gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts243.13' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.626 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0<n(: ) proc.memory.availableMB 2441.27734375) disk./.usagePercent77.29767530003264% disk./.usageGB1151.7912216186523 disk.sda.out 107.828125 disk.drbd1260.out 0.4765625 disk.nvme0n1.in 7948.1953125$ memory_percent85.09962558746338 cpu25.250911346333513 network.sent 4.51231e+06 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0% disk.nvme0c0n1.out82229.60546875 disk.dm-1.in0 disk.drbd1260.in0 disk.dm-0.out82229.60546875 disk.sda3.out 107.828125 disk.sda.in0 disk.dm-1.out 107.828125 disk.sda1.in0 disk.sda1.out0! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.recv 1.10278e+06" disk.nvme0c0n1.in 7948.1953125# disk.nvme0n1.out82229.60546875 disk.dm-0.in 7948.1953125[]8y(: ˸# disk.nvme0c0n1.in 8046.29296875' disk.nvme0c0n1.out82565.5908203125 disk.drbd1260.out 0.48046875% disk.nvme0n1.out82565.5908203125 disk.dm-0.in 8046.29296875 disk.sda3.in0$ memory_percent85.73634624481201) proc.memory.availableMB 2336.95703125) disk./.usagePercent77.30312877252703 disk.drbd1260.in0 disk.sda1.out0 disk.sda3.out 108.265625! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 cpu25.25200055649891 proc.cpu.threads17 network.recv 1.107918e+06% disk./.usageGB1151.8724822998047 disk.dm-1.out 108.265625" disk.dm-0.out82565.5908203125 disk.sda1.in0 disk.sda.in0 disk.sda2.out0 network.sent 4.598208e+06 disk.dm-1.in0! disk.nvme0n1.in 8046.29296875 disk.sda.out 108.265625 disk.sda2.in0wȒ(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts235.895' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.179 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Bs~(:  disk.sda3.in0 disk.sda3.out 108.86328125! proc.memory.rssMB 3122.609375 cpu25.253163181226814$ disk./.usageGB1151.872486114502 disk.dm-1.in0 disk.dm-0.in 8046.29296875 disk.sda.out 108.86328125 disk.sda2.out0# disk.nvme0c0n1.in 8046.29296875 disk.dm-1.out 108.86328125 disk.sda1.out0 disk.sda2.in0# memory_percent86.3971471786499* proc.memory.percent19.058895111083984 proc.cpu.threads17) disk./.usagePercent77.30312902853456' disk.nvme0c0n1.out82660.9306640625 disk.drbd1260.in0 disk.drbd1260.out 0.48046875" disk.dm-0.out82660.9306640625 disk.sda1.in0 disk.sda.in0) proc.memory.availableMB 2228.69140625 network.sent 4.604183e+06 network.recv 1.112508e+06! disk.nvme0n1.in 8046.29296875% disk.nvme0n1.out82660.9306640625>(:  gpu.0.gpu82.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp42.0 gpu.0.powerWatts117.689' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.537799999999997 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0,}(:  disk.drbd1260.in0! disk.nvme0n1.in 8046.29296875& disk.nvme0n1.out83099.06396484375 disk.sda3.out 109.328125 disk.sda.in0 disk.sda2.in0! proc.memory.rssMB 3122.609375 cpu25.254084152546543 network.sent 4.610541e+06 network.recv 1.117343e+06) disk./.usagePercent77.30187587165446 disk.sda1.out0 disk.sda3.in0# disk.nvme0c0n1.in 8046.29296875 disk.drbd1260.out 0.48828125$ memory_percent87.00888156890869# disk.dm-0.out83099.06396484375 disk.sda1.in0( disk.nvme0c0n1.out83099.06396484375) proc.memory.availableMB 2128.46484375* proc.memory.percent19.058895111083984 proc.cpu.threads17% disk./.usageGB1151.8538131713867 disk.dm-0.in 8046.29296875 disk.sda.out 109.328125 disk.sda2.out0 disk.dm-1.in0 disk.dm-1.out 109.3281255(: 䜽 gpu.0.gpu34.0 gpu.0.memory8+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts336.615' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.323 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0FF(: ǻ gpu.0.gpu55.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts208.644' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent41.7288 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0`~(: ǻ( disk.nvme0c0n1.out83327.64599609375 disk.drbd1260.out 0.48828125) proc.memory.availableMB 2016.31640625& disk.nvme0n1.out83327.64599609375 disk.sda2.out0 disk.dm-1.out 109.6640625 disk.drbd1260.in0 proc.cpu.threads17 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.sda2.in0 disk.dm-1.in0$ memory_percent87.69338130950928 cpu25.254900911142563 network.recv 1.122281e+06( disk./.usagePercent77.3018733115791 disk.dm-0.in 8046.29296875# disk.dm-0.out83327.64599609375 disk.sda3.in0 disk.sda.out 109.6640625! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 network.sent 4.616778e+06$ disk./.usageGB1151.853775024414! disk.nvme0n1.in 8046.29296875 disk.sda3.out 109.6640625# disk.nvme0c0n1.in 8046.29296875“(: ֻȱ gpu.0.gpu55.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts317.438' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent63.4876 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Yz(: ֻ$ memory_percent88.35282325744629! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984# disk.dm-0.out83759.40380859375 disk.sda3.in0 disk.sda.in0 disk.sda2.in0 disk.dm-1.in0( proc.memory.availableMB 1908.2734375 cpu25.2558240706283 proc.cpu.threads17 disk.dm-0.in 8175.43359375 disk.sda1.out0 disk.sda3.out 110.0546875# disk.nvme0c0n1.in 8175.43359375 disk.sda.out 110.0546875 network.recv 1.12545e+06) disk./.usagePercent77.30760967239884$ disk./.usageGB1151.939250946045! disk.nvme0n1.in 8175.43359375& disk.nvme0n1.out83759.40380859375( disk.nvme0c0n1.out83759.40380859375 disk.dm-1.out 110.0546875 network.sent 4.621653e+06 disk.sda1.in0 disk.sda2.out0 disk.drbd1260.in0 disk.drbd1260.out 0.4921875w2- _wandbruntime3591  _runtime3591[ & lr-AdamW/pg10.0009840744010971825 & lr-AdamW/pg20.0009840744010971825 trainer/global_step2049 % _timestamp1.7884687013192077e+09 _runtime3591.359786066 _step163Yv trainer/global_step2049 % _timestamp1.7884687013192077e+09 _runtime3591.359786066 _step163 & lr-AdamW/pg10.0009840744010971825 & lr-AdamW/pg20.0009840744010971825%A2- _wandbruntime3591  _runtime3591yO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4753459544064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated29450694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated38505657 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4753459544064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed29450688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated450723726027776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed432579229763588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9077594 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated67956351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed96995100180480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated90923402284544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 trainer/global_step2049 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated445970266483712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4750943180800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated29450694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed445970241055744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed450721184236544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current454 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated14588031 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 6071726993920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed38485973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed38485973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed14587577 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated432581769559968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 _step164 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 # _timestamp1.78846870132046e+09 YEDeviceStatsMonitor.on_train_baQ tch_start/allocated_bytes.all.allocated450723726027776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4750943180800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed67936661 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed29450688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated23665628 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed23665171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current457 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4744703633364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed450721184236544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed445970241055744xs'O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9077594 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated67956351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed96995100180480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated90923402284544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 trainer/global_step2049 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated445970266483712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4750943180800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated29450694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed445970241055744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed450721184236544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current454 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated14588031 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 6071726993920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed38485973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed38485973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed14587577 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated432581769559968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 _step164 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 # _timestamp1.78846870132046e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated450723726027776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4750943180800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed67936661 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed29450688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated23665628 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed23665171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current457 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4744703633364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed450721184236544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed445970241055744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4753459544064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated29450694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated38505657 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4753459544064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed29450688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated450723726027776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 438629888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed432579229763588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0~J\2- _wandbruntime3592  _runtime3592zc&M S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed450934449828352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated446181256859136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed67968893 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed29464569 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6074623390208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated432786570524614 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated97042145970176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated38524034 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed38504324 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed29464569 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed97042057946112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed432784030728026 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed446181231431168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4755734773760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated450936991632896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4753218397184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _step165 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated450936991632896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed38504324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4755734773760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed23676348 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated29464575 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4753218397184 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated38524034 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4749490256838 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated67988609 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current455 $ _timestamp1.788468702797431e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed67968893 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0j%M ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed446181231431168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4755734773760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated450936991632896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4753218397184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 _step165 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated450936991632896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed38504324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 4755734773760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed23676348 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated29464575 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4753218397184 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated38524034 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4749490256838 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated67988609 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current455 $ _timestamp1.788468702797431e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed67968893 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed450934449828352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated446181256859136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed67968893 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed29464569 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6074623390208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated432786570524614 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated97042145970176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated38524034 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed38504324 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed29464569 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed97042057946112 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 M>Devi.EceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed432784030728026 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 2- _wandbruntime3592  _runtime3592?hV 0loss/FBt4/assignment_loss0.1376278685063219 1loss/FBt1/assignment_loss0.12548027234906958 1loss/FBt3/assignment_loss0.10589557253451158 0loss/SRqqt2/detection_loss2.188905904122761 _runtime3592.840097147 /loss/FRt3/assignment_loss3.339101752296823 0loss/FBt2/assignment_loss0.3415132238257105 /loss/FRt3/detection_loss2.8485290759899575 % _timestamp1.7884687027997456e+09 /loss/FBt4/detection_loss2.5841714956543664 trainer/global_step2049 0loss/FRt4/assignment_loss2.4710688355590116 0loss/SRqqt3/detection_loss2.479838237833621 0loss/SRqqt4/detection_loss1.468557210951826 0loss/SRqqt2/assignment_loss1.53908664307424 .loss/FRt2/detection_loss2.693074874141637 2loss/SRqqt3/assignment_loss1.8442060584897426 2loss/SRqqt4/assignment_loss1.7157199570406092 /loss/FRt1/detection_loss1.7680490232693653 1loss/SRqqt1/detection_loss0.9914311403010645 /loss/FBt3/detection_loss2.5331036952825694 epoch3 .loss/FRt1/assignment_loss1.41159954170386 /loss/FRt2/assignment_loss4.090163248426774 2loss/SRqqt1/assignment_loss0.9286500728991615 .loss/FRt4/detection_loss2.018675425849041 /loss/FBt2/detection_loss2.4202913614836605 /loss/FBt1/detection_loss0.6204752726129716 %loss/total_loss42.66521576419868 _step166%[ /loss/FBt1/detection_loss0.6204752726129716 %loss/total_loss42.66521576419868 _step166 0loss/FBt4/assignment_loss0.1376278685063219 1loss/FBt1/assignment_loss0.12548027234906958 1loss/FBt3/assignment_loss0.10589557253451158 0loss/SRqqt2/detection_loss2.188905904122761 _runtime3592.840097147 /loss/FRt3/assignment_loss3.339101752296823 0loss/FBt2/assignment_loss0.3415132238257105 /loss/FRt3/detection_loss2.8485290759899575 % _timestamp1.7884687027997456e+09 /loss/FBt4/detection_loss2.5841714956543664 trainer/global_step2049 0loss/FRt4/assignment_loss2.4710688355590116 0loss/SRqqt3/detection_loss2.479838237833621 0loss/SRqqt4/detection_loss1.468557210951826 0loss/SRqqt2/assignment_loss1.53908664307424 .loss/FRt2/detection_loss2.693074874141637 2loss/SRqqt3/assignment_loss1.8442060584897426 2loss/SRqqt4/assignment_loss1.7157199570406092 /loss/FRt1/detection_loss1.7680490232693653 1loss/SRqqt1/detection_loss0.9914311403010645 /loss/FBt3/detection_loss2.5331036952825694 epoch3 .loss/FRt1/assignment_loss1.41159954170386 /loss/FRt2/assignment_loss4.090163248426774 2loss/SRqqt1/assignment_loss0.9286500728991615 .loss/FRt4/detection_loss2.018675425849041 /loss/FBt2/detection_loss2.4202913614836605 8 }(:  cpu25.256625714445157) disk./.usagePercent77.31180409983982 disk.sda2.out0# disk.nvme0c0n1.in 8175.43359375 disk.drbd1260.in0 disk.sda1.in0 disk.sda3.in0 disk.sda3.out 110.4453125$ memory_percent77.92935371398926 disk.sda2.in0 disk.dm-1.in0! disk.nvme0n1.in 8175.43359375 proc.cpu.threads17( disk.nvme0c0n1.out84033.51708984375 disk.dm-1.out 110.4453125 disk.drbd1260.out 0.4921875# disk.dm-0.out84033.51708984375 disk.sda.out 110.4453125( proc.memory.availableMB 3616.0546875! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 network.sent 4.627654e+06 network.recv 1.129798e+06$ disk./.usageGB1152.001750946045& disk.nvme0n1.out84033.51708984375 disk.dm-0.in 8175.43359375 disk.sda1.out0 disk.sda.in0V (:  gpu.0.gpu80.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts121.16' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent24.232 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0/y(: ' disk.nvme0c0n1.out84285.8486328125) proc.memory.availableMB 3514.12890625! proc.memory.rssMB 3122.609375 cpu25.257398155831826$ disk./.usageGB1152.126750946045! disk.nvme0n1.in 8175.43359375 disk.dm-0.in 8175.43359375 disk.dm-1.in0 disk.dm-1.out 110.921875 disk.drbd1260.in0* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.sent 4.721215e+06) disk./.usagePercent77.32019295472182 disk.sda3.in0 disk.sda.out 110.921875 disk.sda2.in0 network.recv 1.148575e+06 disk.sda1.out0 disk.sda3.out 110.921875 disk.sda2.out0# disk.nvme0c0n1.in 8175.43359375 disk.drbd1260.out 0.49609375$ memory_percent78.55145931243896% disk.nvme0n1.out84285.8486328125" disk.dm-0.out84285.8486328125 disk.sda1.in0 disk.sda.in01M(:  gpu.0.gpu31.0 gpu.0.memory10+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts334.247' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.8494 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0q(:  disk.sda2.out0 disk.sda1.out0* proc.memory.percent19.058895111083984 cpu25.2579294519184 network.recv 1.166561e+06 disk.sda3.in0 disk.sda.in0# disk.nvme0c0n1.in 8175.43359375 disk.dm-1.out 111.34375 disk.drbd1260.out 0.49609375 disk.sda1.in0$ memory_percent79.20341491699219% disk./.usageGB1152.3777694702148 disk.sda3.out 111.34375! disk.nvme0n1.in 8175.43359375% disk.nvme0n1.out84583.9814453125 disk.dm-0.in 8175.43359375" disk.dm-0.out84583.9814453125% proc.memory.availableMB 3407.3125' disk.nvme0c0n1.out84583.9814453125 disk.dm-1.in0 disk.drbd1260.in0! proc.memory.rssMB 3122.609375 proc.cpu.threads17 network.sent 4.734127e+06) disk./.usagePercent77.33703901849742 disk.sda.out 111.34375 disk.sda2.in0;(: נ gpu.0.gpu85.0 gpu.0.memory20+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts333.977' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.7954 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Xl{(: 壟 disk.sda.in0 disk.drbd1260.in0) proc.memory.availableMB 3295.06640625 cpu25.25855567000165 disk.dm-0.in 8290.89453125 disk.sda.out 111.73828125 disk.sda2.out0# disk.nvme0c0n1.in 8290.89453125( disk.nvme0c0n1.out85052.61083984375* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.recv 1.176853e+06# disk.dm-0.out85052.61083984375 disk.dm-1.in0 disk.dm-1.out 111.73828125& disk.nvme0n1.out85052.61083984375$ memory_percent79.88851070404053 network.sent 4.742379e+06 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0 disk.drbd1260.out0.5! disk.nvme0n1.in 8290.89453125! proc.memory.rssMB 3122.609375) disk./.usagePercent77.33542463498651% disk./.usageGB1152.3537139892578 disk.sda3.out 111.73828125(:  gpu.0.gpu52.0 gpu.0.memory12+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts159.536' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent31.907200000000003 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0v&:(: Ӌ gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts161.994' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent32.3988 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0U͵q(: ш! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 proc.cpu.threads17% disk./.usageGB1152.3771896362305 disk.dm-0.in 8290.89453125 disk.sda2.out0% disk.nvme0c0n1.out85397.39453125 disk.dm-1.in0 disk.dm-1.out 112.05078125 disk.drbd1260.in0 disk.drbd1260.out0.5 disk.dm-0.out85397.39453125 disk.sda1.in0 disk.sda3.in0 disk.sda3.out 112.05078125# disk.nvme0c0n1.in 8290.89453125' proc.memory.availableMB 3186.796875 network.sent 4.749523e+06 network.recv 1.182539e+06# disk.nvme0n1.out85397.39453125 disk.sda1.out0 disk.sda.in0 disk.sda.out 112.05078125 disk.sda2.in0 cpu25.259168909229764) disk./.usagePercent77.33700010535222! disk.nvme0n1.in 8290.89453125$ memory_percent80.54933547973633v2- _wandbruntime3674  _runtime3674 & lr-AdamW/pg10.0009712959201668793 & lr-AdamW/pg20.0009712959201668793 trainer/global_step2099 % _timestamp1.7884687848481932e+09 _runtime3674.888763649 _step167rY % _timestamp1.7884687848481932e+09 _runtime3674.888763649 _step167 & lr-AdamW/pg10.0009712959201668793 & lr-AdamW/pg20.0009712959201668793 trainer/global_step2099^2- _wandbruntime3674  _runtime3674҄'O TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated461387033446400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed93126441653760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _step168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed442819305157496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated39426559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed39406875 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9294194 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated24227579 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed456519759826944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed461384491655168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4864731828224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated69571303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated30144744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4864731828224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated456519785254912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed30144738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 _runtime3674.890757489 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673岔'O =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1384119808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated461387033446400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed93126441653760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 _step168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed442819305157496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated39426559 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed39406875 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9294194 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated24227579 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed456519759826944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed461384491655168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4864731828224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated69571303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated30144744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 4864731828224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated456519785254912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed30144738 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 _runtime3674.890757489 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527.(:  gpu.0.gpu63.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp47.0 gpu.0.powerWatts178.953' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent35.7906 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors04Uy(:  disk.sda3.in0# disk.nvme0c0n1.in 8290.89453125$ memory_percent81.18553161621094% proc.memory.availableMB 3082.5625% disk./.usageGB1152.3771896362305 disk.dm-1.out 112.6328125 disk.drbd1260.in0% disk.nvme0n1.out85805.2216796875 disk.sda3.out 112.6328125 disk.sda2.in0! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 disk.drbd1260.out 0.50390625 disk.dm-0.in 8290.89453125 cpu25.259958286790138 proc.cpu.threads17) disk./.usagePercent77.33700010535222" disk.dm-0.out85805.2216796875 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.sda.out 112.6328125 disk.sda2.out0' disk.nvme0c0n1.out85805.2216796875 network.sent 4.755092e+06 network.recv 1.186783e+06 disk.dm-1.in0! disk.nvme0n1.in 8290.89453125nQ(2- _wandbruntime3676  _runtime3676x9M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed30158619 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed461597757246976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated24238852 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated30158625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 3706087 IϠ&$0144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4860615802926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated461600299051520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated456730775630336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed456730750202368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed93170503023104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated461600299051520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed456730750202368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed443024106121934 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 % _timestamp1.7884687863253467e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed69583845 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed24238379 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated456730775630336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated30158625 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed39425226 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated39444936 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 _step169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed69583845 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9298526 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed39425226 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime3676.366559349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed30158619 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated39444936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4869523421184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated69603561 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed438163490319008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current470 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4867007044608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated69603561 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated14940323 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current473 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4867007044608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6219734863872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4863130171546 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated9298529 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6219675937792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146RVR'M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated456730775630336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed456730750202368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed93170503023104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated461600299051520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed456730750202368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed443024106121934 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 % _timestamp1.7884687863253467e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed69583845 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed24238379 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated456730775630336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated30158625 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed39425226 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated39444936 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 _step169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed69583845 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9298526 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed39425226 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 438629888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 _runtime3676.366559349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed30158619 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated39444936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 4869523421184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated69603561 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed438163490319008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current470 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1384119808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 4867007044608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated69603561 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated14940323 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current473 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 4867007044608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6219734863872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4863130171546 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated9298529 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6219675937792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed30158619 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed461597757246976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated24238852 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated30158625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4860615802926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated461600299051520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376b2- _wandbruntime3676  _runtime3676qs\ 0loss/FRt1/assignment_loss1.5685233955431466 1loss/FBt1/assignment_loss0.14965630451907525 /loss/FBt3/detection_loss2.6866789428811324 &loss/total_loss46.169738513968674 _step170 /loss/FRt3/assignment_loss3.612257258594036 /loss/FRt4/assignment_loss2.993795615490352 2loss/SRqqt2/assignment_loss2.0092870336575586 1loss/SRqqt4/assignment_loss1.752759907367847 1loss/FBt2/assignment_loss0.15070819769753144 /loss/FRt2/detection_loss2.7882331157552787 /loss/FBt1/detection_loss0.6348462766845024 trainer/global_step2099 /loss/FRt3/detection_loss2.6077211938798426 1loss/SRqqt3/detection_loss2.2330579217523336 /loss/FBt4/detection_loss2.7800975075134864 2loss/SRqqt1/assignment_loss0.9778615501360829 1loss/SRqqt3/assignment_loss2.014633549377322 /loss/FRt1/detection_loss1.8113100316488382 1loss/SRqqt4/detection_loss1.6131254386344402 /loss/FBt2/detection_loss2.5201145499944686 _runtime3676.368809201 /loss/FRt2/assignment_loss4.362233612044104 1loss/SRqqt1/detection_loss0.9023315532981944 % _timestamp1.7884687863283088e+09 1loss/FBt3/assignment_loss0.23027040828693365 /loss/FBt4/assignment_loss0.892693443421964 /loss/FRt4/detection_loss2.2670739298668976 0loss/SRqqt2/detection_loss2.610467775923307 epoch3a /loss/FBt4/assignment_loss0.892693443421964 /loss/FRt4/detection_loss2.2670739298668976 0loss/SRqqt2/detection_loss2.610467775923307 epoch3 0loss/FRt1/assignment_loss1.5685233955431466 1loss/FBt1/assignment_loss0.14965630451907525 /loss/FBt3/detection_loss2.6866789428811324 &loss/total_loss46.169738513968674 _step170 /loss/FRt3/assignment_loss3.612257258594036 /loss/FRt4/assignment_loss2.993795615490352 2loss/SRqqt2/assignment_loss2.0092870336575586 1loss/SRqqt4/assignment_loss1.752759907367847 1loss/FBt2/assignment_loss0.15070819769753144 /loss/FRt2/detection_loss2.7882331157552787 /loss/FBt1/detection_loss0.6348462766845024 trainer/global_step2099 /loss/FRt3/detection_loss2.6077211938798426 1loss/SRqqt3/detection_loss2.2330579217523336 /loss/FBt4/detection_loss2.7800975075134864 2loss/SRqqt1/assignment_loss0.9778615501360829 1loss/SRqqt3/assignment_loss2.014633549377322 /loss/FRt1/detection_loss1.8113100316488382 1loss/SRqqt4/detection_loss1.6131254386344402 /loss/FBt2/detection_loss2.5201145499944686 _runtime3676.368809201 /loss/FRt2/assignment_loss4.362233612044104 1loss/SRqqt1/detection_loss0.9023315532981944 % _timestamp1.7884687863283088e+09 1loss/FBt3/assignment_loss0.23027040828693365O t(: ) proc.memory.availableMB 2978.28515625! proc.memory.rssMB 3122.609375 network.sent 4.830698e+06 disk.sda3.in0 disk.sda2.in0 proc.cpu.threads17 network.recv 1.192403e+06% disk./.usageGB1152.8773078918457 disk.sda.in0 disk.sda2.out0# disk.nvme0c0n1.in 8290.89453125$ memory_percent81.82199001312256* proc.memory.percent19.058895111083984 cpu25.260384362095305) disk./.usagePercent77.37056346111373! disk.nvme0n1.in 8290.89453125 disk.dm-0.in 8290.89453125 disk.sda1.out0 disk.sda3.out113.0625' disk.nvme0c0n1.out85959.2216796875 disk.drbd1260.in0 disk.drbd1260.out 0.50390625% disk.nvme0n1.out85959.2216796875" disk.dm-0.out85959.2216796875 disk.sda1.in0 disk.sda.out113.0625 disk.dm-1.in0 disk.dm-1.out113.0625%@n(:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts237.238' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.4476 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 @:t(: μߨ disk.dm-1.in0 disk.sda3.in0 disk.sda2.out0! proc.memory.rssMB 3122.609375) disk./.usagePercent77.27142966362986% disk./.usageGB1151.4001426696777 disk.dm-1.out 113.54296875 disk.drbd1260.in0 disk.nvme0n1.in 8460.203125* proc.memory.percent19.058895111083984 cpu25.261562898969842 proc.cpu.threads17 network.sent 4.906528e+06 network.recv 1.197269e+06& disk.nvme0c0n1.out86258.759765625 disk.dm-0.in 8460.203125! disk.dm-0.out86258.759765625 disk.sda3.out 113.54296875 disk.sda.in0$ memory_percent82.45782852172852! disk.nvme0c0n1.in 8460.203125 disk.drbd1260.out 0.5078125$ disk.nvme0n1.out86258.759765625 disk.sda1.in0 disk.sda1.out0 disk.sda.out 113.54296875 disk.sda2.in0' proc.memory.availableMB 2874.109375qޑ(: μ gpu.0.gpu74.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts332.53' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.506 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors00J(: ݼ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp42.0 gpu.0.powerWatts178.264' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent35.6528 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0@Yt(: ݼ! proc.memory.rssMB 3122.609375) disk./.usagePercent77.28194517309878 disk.sda1.in0 disk.sda2.in0! disk.nvme0c0n1.in 8460.203125& disk.nvme0c0n1.out86945.486328125! disk.dm-0.out86945.486328125$ memory_percent83.07714462280273* proc.memory.percent19.058895111083984 network.sent 4.922302e+06% disk./.usageGB1151.5568313598633 disk.sda1.out0 disk.sda.out 113.91015625 disk.dm-1.out 113.91015625 disk.drbd1260.out 0.5078125 cpu25.262387495166035 proc.cpu.threads17 disk.sda3.in0 disk.sda3.out 113.91015625 disk.sda2.out0 disk.drbd1260.in0$ disk.nvme0n1.out86945.486328125 disk.dm-0.in 8460.203125 network.recv 1.221882e+06 disk.sda.in0 disk.dm-1.in0 disk.nvme0n1.in 8460.203125' proc.memory.availableMB 2772.640625@P(:  gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp49.0 gpu.0.powerWatts329.179' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.8358 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0t(:  disk.dm-1.out 114.421875 disk.drbd1260.out 0.51171875 disk.dm-0.in 8460.203125 disk.sda3.in0 disk.sda3.out 114.421875 disk.sda.in0 disk.sda.out 114.421875! disk.nvme0c0n1.in 8460.203125 disk.dm-1.in0 disk.drbd1260.in0 disk.nvme0n1.in 8460.203125% disk.nvme0n1.out87173.1787109375" disk.dm-0.out87173.1787109375 disk.sda1.out0$ memory_percent83.70912075042725! proc.memory.rssMB 3122.609375 network.recv 1.226098e+06 disk.sda2.in0 disk.sda2.out0' disk.nvme0c0n1.out87173.1787109375* proc.memory.percent19.058895111083984 cpu25.263269057149195 network.sent 4.927851e+06) disk./.usagePercent77.28374669811723% disk./.usageGB1151.5836753845215 disk.sda1.in0) proc.memory.availableMB 2669.09765625 proc.cpu.threads17-t(:  disk.nvme0n1.in 8460.203125" disk.dm-0.out87548.8115234375 disk.sda3.in0 disk.sda2.out0 disk.dm-1.in0 disk.drbd1260.out 0.51171875* proc.memory.percent19.058895111083984 network.sent 4.933597e+06 network.recv 1.230396e+06$ disk./.usageGB1151.583122253418 disk.dm-0.in 8460.203125 disk.sda1.in0 disk.sda.out 115.08203125 disk.sda2.in0 disk.dm-1.out 115.08203125 disk.sda1.out0 disk.sda3.out 115.08203125 disk.sda.in0! disk.nvme0c0n1.in 8460.203125' disk.nvme0c0n1.out87548.8115234375 disk.drbd1260.in0$ memory_percent84.39445495605469% proc.memory.availableMB 2556.8125) disk./.usagePercent77.28370957702477% disk.nvme0n1.out87548.8115234375! proc.memory.rssMB 3122.609375 cpu25.26411218036918 proc.cpu.threads17ܑ7v(: 疵 gpu.0.gpu85.0 gpu.0.memory20+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts325.434' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.0868 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors02- _wandbruntime3758  _runtime3758d _step171 % lr-AdamW/pg10.000958198145675873 % lr-AdamW/pg20.000958198145675873 trainer/global_step2149 % _timestamp1.7884688681936767e+09 _runtime3758.234429817v. % lr-AdamW/pg10.000958198145675873 % lr-AdamW/pg20.000958198145675873 trainer/global_step2149 % _timestamp1.7884688681936767e+09 _runtime3758.234429817 _step171/2- _wandbruntime3758  _runtime3758aKIO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed472047799073792 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current454 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated9510797 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 6362010848768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed448087397088624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated40347461 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed30838788 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current451 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated30838794 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed24789664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9510794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated453061920347784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 440866304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4974497831192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated467069304026112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated448087422516592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4981036838912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed472047799073792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed467069278598144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4971983462780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated472050340865024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 _runtime3758.236392328 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated71186255 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated101691609007104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated472050340865024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed40327777 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4978520475648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6362069788160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated467069304026112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated40347461 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed40327777 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated95329539218944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 trainer/global_step2149 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed71166565 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed467069278598144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated15279321 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4981036838912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144vL'O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated15279321 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 4981036838912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed472047799073792 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current454 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated9510797 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 6362010848768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed448087397088624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated40347461 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed30838788 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current451 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated30838794 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed24789664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9510794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated453061920347784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 440866304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 4974497831192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated467069304026112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated448087422516592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 4981036838912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed472047799073792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed467069278598144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 4971983462780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated472050340865024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 _runtime3758.236392328 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated71186255 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated101691609007104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated472050340865024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed40327777 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 4978520475648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6362069788160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated467069304026112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated40347461 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed40327777 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated95329539218944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 trainer/global_step2149 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed71166565 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed467069278598144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 4026322124812- _wandbruntime3759  _runtime3759I&M WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated472263606470144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated40365838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15285684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated95373600588288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed448289925798208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed24800810 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 $ _timestamp1.788468869692707e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated71218513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated30852675 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current471 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1388313088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed30852669 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed467280268973568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9515126 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6364923061760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed101738494552064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed71198797 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated453266721312430 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated9515129 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed71198797 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4974255717634 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed40346128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated472263606470144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4976770086254 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 trainer/global_step2149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated467280294401536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248itM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated467280294401536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated71218513 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated30852675 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current471 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1388313088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed30852669 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed467280268973568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9515126 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6364923061760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed101738494552064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed71198797 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated453266721312430 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated9515129 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed71198797 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 4974255717634 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed40346128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated472263606470144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 4976770086254 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 trainer/global_step2149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated467280294401536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated472263606470144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated40365838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15285684 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated95373600588288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed448289925798208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed24800810 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 $ _timestamp1.788468869692707e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current458 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed104039777013248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 trainer/global_step2199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed41248679 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed458213832567824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated458213857995792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed482711106492416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 _runtime 3840.81352424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated72801207 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated25352118 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5085623377488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed31532838 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5094825486336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5088137745900 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated477618822797312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed482711106492416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed477618797369344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed31532838 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated31532844 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated9727397 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed477618797369344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5094825486336 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90134528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9727394 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated482713648283648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated482713648283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated97532607686144Oa9O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current458 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3673 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3673 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed104039777013248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 trainer/global_step2199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed41248679 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed458213832567824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated458213857995792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3202351104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed482711106492416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 _runtime 3840.81352424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated72801207 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated25352118 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5085623377488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak879 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed31532838 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5094825486336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5088137745900 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated477618822797312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 UADeviceStatsMonitor.o;٧ n_train_batch_start/allocated_bytes.all.freed482711106492416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40263221248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed477618797369344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed31532838 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated31532844 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated9727397 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed477618797369344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5094825486336 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90134528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9727394 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated482713648283648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated482713648283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated97532607686144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5092309123072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed97532578588160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated104039867147776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 6507198425088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 % _timestamp1.7884689507712803e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed15624263 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated463301995741692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak2368262- _wandbruntime3842  _runtime3842Iz&M ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6510096837632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed458416361277408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5090410000962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed41267030 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed97576639957504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5087895632342 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5097100716032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed104086736795136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated41286740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated477829813172736 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90121216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed41267030 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed72813749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15631101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated97576669055488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6510157860864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated31546725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current468 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed25362827 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step2199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 _step177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated477829813172736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 440866304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9731726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated72833465 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed31546719 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed482924372084224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5094584339456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed31546719 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated104086826916352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5097100716032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed41267030 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3202351104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1527 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed97576639957504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5087895632342 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5097100716032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed104086736795136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated41286740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated477829813172736 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90121216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed41267030 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed72813749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15631101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated97576669055488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6510157860864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated31546725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current468 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed25362827 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 trainer/global_step2199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 _step177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated477829813172736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 440866304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40263221248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9731726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated72833465 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed31546719 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed482924372084224C\2- _wandbruntime3842  _runtime3842PVY 1loss/FBt1/assignment_loss0.18272064825501053 0loss/FRt4/assignment_loss3.5480012612522773 2loss/SRqqt1/assignment_loss1.0170197862832409 2loss/SRqqt3/assignment_loss2.0680070600339344 /loss/FBt2/detection_loss2.7991005040350414 .loss/FBt3/detection_loss3.187150614602225 _runtime 3842.33089086 1loss/FBt2/assignment_loss0.29789305015306344 0loss/SRqqt4/detection_loss1.728848613023758 trainer/global_step2199 0loss/FBt3/assignment_loss0.7688829034914739 1loss/FBt4/assignment_loss0.14179155270961163 .loss/FRt2/detection_loss2.628295522846588 0loss/FRt1/assignment_loss1.2993102669715881 0loss/FRt3/assignment_loss3.5378376726181275 /loss/FRt3/detection_loss2.8377328918826197 /loss/FBt1/detection_loss0.7693005862084697 /loss/FRt2/assignment_loss4.061985899324286 2loss/SRqqt4/assignment_loss1.8997253468632698 %loss/total_loss47.04556923898863 /loss/FRt1/detection_loss1.6864648257883696 epoch3 _step178 1loss/SRqqt2/detection_loss2.2345083009552313 1loss/SRqqt3/detection_loss2.3571211533887046 % _timestamp1.7884689522905402e+09 0loss/SRqqt2/assignment_loss1.97952196787338 1loss/SRqqt1/detection_loss0.9834926162447248 .loss/FRt4/detection_loss2.165543153600873 .loss/FBt4/detection_loss2.865313040582757)S^ /loss/FRt1/detection_loss1.6864648257883696 epoch3 _step178 1loss/SRqqt2/detection_loss2.2345083009552313 1loss/SRqqt3/detection_loss2.3571211533887046 % _timestamp1.7884689522905402e+09 0loss/SRqqt2/assignment_loss1.97952196787338 1loss/SRqqt1/detection_loss0.9834926162447248 .loss/FRt4/detection_loss2.165543153600873 .loss/FBt4/detection_loss2.865313040582757 1loss/FBt1/assignment_loss0.18272064825501053 0loss/FRt4/assignment_loss3.5480012612522773 2loss/SRqqt1/assignment_loss1.0170197862832409 2loss/SRqqt3/assignment_loss2.0680070600339344 /loss/FBt2/detection_loss2.7991005040350414 .loss/FBt3/detection_loss3.187150614602225 _runtime 3842.33089086 1loss/FBt2/assignment_loss0.29789305015306344 0loss/SRqqt4/detection_loss1.728848613023758 trainer/global_step2199 0loss/FBt3/assignment_loss0.7688829034914739 1loss/FBt4/assignment_loss0.14179155270961163 .loss/FRt2/detection_loss2.628295522846588 0loss/FRt1/assignment_loss1.2993102669715881 0loss/FRt3/assignment_loss3.5378376726181275 /loss/FRt3/detection_loss2.8377328918826197 /loss/FBt1/detection_loss0.7693005862084697 /loss/FRt2/assignment_loss4.061985899324286 2loss/SRqqt4/assignment_loss1.8997253468632698 %loss/total_loss47.04556923898863[t(: % disk./.usageGB1150.3525009155273 disk.drbd1260.in0 disk.sda.in0 disk.sda2.out0 cpu25.26821508730312 proc.cpu.threads17# disk.nvme0c0n1.in 8760.23046875( disk.nvme0c0n1.out89184.26611328125 disk.dm-1.out118.4375 disk.drbd1260.out 0.52734375# disk.dm-0.out89184.26611328125 disk.sda1.in0 disk.sda2.in0# memory_percent88.9289379119873 network.sent 5.053061e+06 disk.dm-1.in0! disk.nvme0n1.in 8760.23046875& disk.nvme0n1.out89184.26611328125 disk.sda.out118.4375( proc.memory.availableMB 1813.8828125! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 network.recv 1.259888e+06) disk./.usagePercent77.20112154647884 disk.dm-0.in 8760.23046875 disk.sda1.out0 disk.sda3.in0 disk.sda3.out118.4375u(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts231.639' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.3278 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 >)(:  gpu.0.gpu69.0 gpu.0.memory14+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts247.411' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.4822 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors00M(:  disk.sda2.out0# disk.nvme0c0n1.in 8760.23046875 disk.drbd1260.in0% disk./.usageGB1150.5075950622559 disk.sda2.in0 disk.dm-1.in0# memory_percent88.9275312423706! proc.memory.rssMB 3122.609375 cpu25.268681784487832) disk./.usagePercent77.21153004479845& disk.nvme0n1.out89367.53466796875 disk.dm-0.in 8760.23046875( disk.nvme0c0n1.out89367.53466796875 disk.dm-1.out 118.93359375) proc.memory.availableMB 1814.11328125 network.recv 1.265803e+06 disk.sda1.out0 disk.sda3.out 118.93359375 disk.sda.in0 disk.sda.out 118.93359375 disk.drbd1260.out 0.52734375* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.sent 5.14015e+06! disk.nvme0n1.in 8760.23046875# disk.dm-0.out89367.53466796875 disk.sda1.in0 disk.sda3.in07z(:  network.sent 5.14521e+06) disk./.usagePercent77.22193854311806% disk./.usageGB1150.6626892089844 disk.dm-1.out 119.3984375 disk.dm-1.in0 disk.drbd1260.in0 disk.sda3.out 119.3984375 disk.sda2.out0( proc.memory.availableMB 1814.1171875* proc.memory.percent19.058895111083984 cpu25.269377749183295 proc.cpu.threads17& disk.nvme0n1.out89937.85107421875 disk.dm-0.in 8760.23046875 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0# disk.nvme0c0n1.in 8760.23046875# memory_percent88.9275074005127 network.recv 1.269336e+06 disk.drbd1260.out0.53125! disk.nvme0n1.in 8760.23046875# disk.dm-0.out89937.85107421875 disk.sda.in0 disk.sda.out 119.3984375( disk.nvme0c0n1.out89937.85107421875! proc.memory.rssMB 3122.609375(:  gpu.0.gpu64.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts313.386' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent62.6772 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0BMǒ(: Ǟ gpu.0.gpu96.0 gpu.0.memory19+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts331.285' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.257 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0qt(:  network.sent 5.151354e+06% disk./.usageGB1150.5381927490234 disk.dm-1.out 119.76171875& disk.nvme0n1.out90131.46435546875 disk.dm-0.in 8871.265625 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.dm-1.in0 disk.nvme0n1.in 8871.265625 disk.sda3.in0 disk.sda.out 119.76171875 disk.sda2.in0 disk.sda2.out0$ memory_percent88.93203735351562! proc.memory.rssMB 3122.609375 disk.drbd1260.out0.53125# disk.dm-0.out90131.46435546875 disk.sda3.out 119.76171875$ proc.memory.availableMB1813.375 proc.cpu.threads17 network.recv 1.274024e+06( disk./.usagePercent77.2135834812306! disk.nvme0c0n1.in 8871.265625( disk.nvme0c0n1.out90131.46435546875 disk.drbd1260.in0* proc.memory.percent19.058895111083984 cpu25.270040038642286ot(: ֝ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts246.885' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.376999999999995 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0s{(:  disk.sda1.out0 disk.sda3.in0 disk.sda.in0( disk.nvme0c0n1.out90422.85205078125 disk.dm-1.out 120.17578125 disk.drbd1260.out 0.53515625 network.sent 5.15676e+06) disk./.usagePercent77.21516509577712 disk.sda.out 120.17578125 disk.sda2.out0! disk.nvme0c0n1.in 8871.265625 disk.dm-1.in0 disk.drbd1260.in0$ memory_percent88.92395496368408 disk.sda1.in0 disk.sda2.in0 disk.dm-0.in 8871.265625! proc.memory.rssMB 3122.609375 cpu25.27058458458869 proc.cpu.threads17 disk.sda3.out 120.17578125 disk.nvme0n1.in 8871.265625& disk.nvme0n1.out90422.85205078125# disk.dm-0.out90422.85205078125) proc.memory.availableMB 1814.69921875* proc.memory.percent19.058895111083984 network.recv 1.277974e+06% disk./.usageGB1150.5617599487305)JZ2- _wandbruntime3924  _runtime3924^V & lr-AdamW/pg10.0009310887164263664 & lr-AdamW/pg20.0009310887164263664 trainer/global_step2249 $ _timestamp1.788469034943579e+09 _runtime3924.984375233 _step179m,W & lr-AdamW/pg20.0009310887164263664 trainer/global_step2249 $ _timestg*ltamp1.788469034943579e+09 _runtime3924.984375233 _step179 & lr-AdamW/pg10.0009310887164263664& @2- _wandbruntime3924  _runtime3924À'O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated493376955702272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated42189265 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 $ _timestamp1.788469034945305e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated493376955702272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed15968050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated99735676153344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated473542071135600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated468340293474992 _runtime3924.986356569 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5206097770496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed42169581 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated25912510 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated32226894 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6652360273920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed468340268047024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed493374413911040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated15968513 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed42169581 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated488168341568512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed488168316140544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current466 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current463 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 440866304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed488168316140544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed32226888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed473539531339220 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 trainer/global_step2249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9943994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5208614133760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5206097770496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed32226888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated74416159 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed74396469 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated32226894 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated42189265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed493374413911040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed25912044 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated106388036427264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed106387948389888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed99735647055360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682X)'O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5206097770496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40263221248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1527 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed32226888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated74416159 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed74396469 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated32226894 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3202351104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated42189265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed493374413911040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed25912044 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated106388036427264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3202351104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed106387948389888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed99735647055360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated493376955702272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated42189265 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 $ _timestamp1.788469034945305e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated493376955702272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed15968050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated99735676153344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated473542071135600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated468340293474992 _runtime3924.986356569 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5206097770496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40263221248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed42169581 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated25912510 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3673 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1527 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated32226894 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6652360273920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed468340268047024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1527 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed493374413911040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated15968513 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed42169581 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated488168341568512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed488168316140544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current466 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current463 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 440866304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed488168316140544 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed32226888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1388313088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed473539531339220 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 trainer/global_step2249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed9943994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5208614133760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146Ŭ2- _wandbruntime3926  _runtime3926~A&M K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed493587679502848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5201535547050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated32240775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 440866304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5208372986880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated106435020144128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated74448417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed32240769 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed488379306515968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed74428701 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5204049915670 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5210889363456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated32240775 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1388313088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed74428701 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed488379306515968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed473744332303658 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9948326 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed106434932120064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated473746872100246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed42187932 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed468542796756608 trainer/global_step2249 % _timestamp1.7884690364238544e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed42187932 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated25923729 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak879 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated468542822184576 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed25923270 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed32240769 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated42207642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15974944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated99779737522688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6655223695360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5210889363456 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms07 M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3202351104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed25923270 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed32240769 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated42207642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed15974944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated99779737522688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6655223695360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batchq]_end/allocated_bytes.small_pool.allocated 5210889363456 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed493587679502848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5201535547050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated32240775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 440866304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5208372986880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated106435020144128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40263221248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3673 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated74448417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed32240769 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed488379306515968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed74428701 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5204049915670 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current459 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5210889363456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated32240775 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1388313088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed74428701 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed488379306515968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed473744332303658 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3202351104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed9948326 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed106434932120064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated473746872100246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40263221248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed42187932 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed468542796756608 trainer/global_step2249 % _timestamp1.7884690364238544e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed42187932 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated25923729 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak879 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1527 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated468542822184576 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak3285P2- _wandbruntime3926  _runtime3926kR -loss/FRt3/detection_loss2.98075026593038 2loss/SRqqt1/assignment_loss0.9652828587820446 1loss/SRqqt3/assignment_loss2.092857207990054 1loss/FBt1/assignment_loss0.20557704217909448 -loss/FRt2/detection_loss2.43637235738613 1loss/SRqqt4/detection_loss1.6376500168016979 .loss/FBt2/detection_loss5.020611842473348 epoch3 1loss/SRqqt4/assignment_loss2.033853156704988 /loss/FBt3/assignment_loss0.539485065639019 .loss/FBt3/detection_loss4.155055046081543 .loss/FBt4/detection_loss2.385959663242102 1loss/FBt2/assignment_loss0.25973358810491237 0loss/FBt4/assignment_loss0.8191908238455653 %loss/total_loss50.54057691231715 1loss/SRqqt2/detection_loss2.4179038058405053 trainer/global_step2249 1loss/SRqqt1/detection_loss1.0692460352158921 .loss/FBt1/detection_loss0.739695230772486 _runtime3926.466490979 0loss/FRt2/assignment_loss3.8457281405543102 /loss/FRt4/assignment_loss3.231842712064584 1loss/SRqqt3/detection_loss2.4946898434198266 % _timestamp1.7884690364261897e+09 /loss/FRt3/assignment_loss3.504089960030147 /loss/FRt1/detection_loss1.8539191775924557 /loss/FRt4/detection_loss2.1275249513487022 _step182 /loss/FRt1/assignment_loss1.700394976925064 1loss/SRqqt2/assignment_loss2.023163143392295s8W /loss/FRt3/assignment_loss3.504089960030147 /loss/FRt1/detection_loss1.8539191775924557 /loss/FRt4/detection_loss2.1275249513487022 _step182 /loss/FRt1/assignment_loss1.700394976925064 1loss/SRqqt2/assignment_loss2.023163143392295 -loss/FRt3/detection_loss2.98075026593038 2loss/SRqqt1/assignment_loss0.9652828587820446 1loss/SRqqt3/assignment_loss2.092857207990054 1loss/FBt1/assignment_loss0.20557704217909448 -loss/FRt2/detection_loss2.43637235738613 1loss/SRqqt4/detection_loss1.6376500168016979 .loss/FBt2/detection_loss5.020611842473348 epoch3 1loss/SRqqt4/assignment_loss2.033853156704988 /loss/FBt3/assignment_loss0.539485065639019 .loss/FBt3/detection_loss4.155055046081543 .loss/FBt4/detection_loss2.385959663242102 1loss/FBt2/assignment_loss0.25973358810491237 0loss/FBt4/assignment_loss0.8191908238455653 %loss/total_loss50.54057691231715 1loss/SRqqt2/detection_loss2.4179038058405053 trainer/global_step2249 1loss/SRqqt1/detection_loss1.0692460352158921 .loss/FBt1/detection_loss0.739695230772486 _runtime3926.466490979 0loss/FRt2/assignment_loss3.8457281405543102 /loss/FRt4/assignment_loss3.231842712064584 1loss/SRqqt3/detection_loss2.4946898434198266 % _timestamp1.7884690364261897e+09;w(:  network.recv 1.282033e+06% disk./.usageGB1150.7253341674805 disk.sda.out 120.6015625 disk.sda2.in0! disk.nvme0c0n1.in 8871.265625 disk.dm-0.in 8871.265625 disk.sda1.out0# memory_percent88.9258861541748 network.sent 5.17555e+06 disk.sda2.out0& disk.nvme0n1.out90661.73486328125( proc.memory.availableMB 1814.3828125 cpu25.271063633563095 disk.sda3.in0 disk.sda.in0 disk.dm-1.in0 disk.dm-1.out 120.6015625! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 proc.cpu.threads17) disk./.usagePercent77.22614269884535 disk.sda3.out 120.6015625( disk.nvme0c0n1.out90661.73486328125 disk.drbd1260.in0 disk.drbd1260.out 0.53515625 disk.nvme0n1.in 8871.265625# disk.dm-0.out90661.73486328125 disk.sda1.in0X (:  gpu.0.gpu99.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp43.0 gpu.0.powerWatts120.341' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent24.068199999999997 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0+$M-u(: ڏ# disk.dm-0.out90767.39013671875 disk.sda3.in0 disk.sda.in0 disk.sda2.out0! disk.nvme0c0n1.in 8871.265625 network.sent 5.24885e+06 disk.sda1.in0 disk.sda3.out 121.23046875 disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.out 0.5390625$ proc.memory.availableMB1814.625* proc.memory.percent19.058895111083984) disk./.usagePercent77.22666495421521 disk.sda.out 121.23046875( disk.nvme0c0n1.out90767.39013671875$ memory_percent88.92440795898438 proc.cpu.threads17 network.recv 1.287278e+06% disk./.usageGB1150.7331161499023 disk.sda1.out0 disk.dm-1.out 121.23046875 disk.drbd1260.in0! proc.memory.rssMB 3122.609375 cpu25.27187964378682 disk.nvme0n1.in 8871.265625& disk.nvme0n1.out90767.39013671875 disk.dm-0.in 8871.265625':a(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp46.0 gpu.0.powerWatts308.917' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent61.7834 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0s`<(: ; gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp40.0 gpu.0.powerWatts115.521' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.1042 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ˉ-|(: ;͌# memory_percent88.9244794845581) proc.memory.availableMB 1814.61328125 cpu25.27222501117647 network.sent 5.256683e+06 network.recv 1.29177e+06) disk./.usagePercent77.22540411710906 disk.drbd1260.in0! disk.nvme0n1.in 9007.83984375 disk.sda.out 121.5390625* proc.memory.percent19.058895111083984 disk.drbd1260.out 0.5390625# disk.dm-0.out90964.92919921875 disk.sda1.out0 disk.sda2.in0 disk.dm-1.in0 disk.dm-1.out 121.5390625 proc.cpu.threads17% disk./.usageGB1150.7143287658691 disk.dm-0.in 9007.83984375 disk.sda1.in0 disk.sda3.in0# disk.nvme0c0n1.in 9007.83984375! proc.memory.rssMB 3122.609375& disk.nvme0n1.out90964.92919921875 disk.sda3.out 121.5390625 disk.sda.in0 disk.sda2.out0( disk.nvme0c0n1.out90964.92919921875X?(: ܾ gpu.0.gpu97.0 gpu.0.memory21+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp48.0 gpu.0.powerWatts325.962' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent65.19239999999999 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0}g(: ܾ disk.sda3.out121.875 disk.drbd1260.out 0.54296875! disk.nvme0n1.in 9007.83984375* proc.memory.percent19.058895111083984 disk.sda1.out0 disk.sda.in0 disk.sda2.in0 disk.sda2.out0# disk.nvme0c0n1.in 9007.83984375% disk.nvme0c0n1.out91255.19921875 disk.dm-1.out121.875 cpu25.27305441932048 network.recv 1.295962e+06 disk.sda.out121.875 disk.dm-1.in0# disk.nvme0n1.out91255.19921875$ memory_percent88.92602920532227' proc.memory.availableMB 1814.359375! proc.memory.rssMB 3122.609375( disk./.usagePercent77.2254043731166% disk./.usageGB1150.7143325805664 disk.sda1.in0 disk.sda3.in0 disk.drbd1260.in0 disk.dm-0.in 9007.83984375 disk.dm-0.out91255.19921875 proc.cpu.threads17 network.sent 5.260486e+06t(: % disk./.usageGB1150.9017143249512# disk.nvme0n1.out91698.52734375 disk.sda3.out 122.421875# disk.nvme0c0n1.in 9007.83984375 disk.dm-1.in0 disk.drbd1260.out 0.54296875$ memory_percent88.92452716827393 disk.dm-0.out91698.52734375 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda.out 122.421875 disk.dm-1.out 122.421875 disk.drbd1260.in0 network.sent 5.268073e+06! disk.nvme0n1.in 9007.83984375 disk.sda2.in0) proc.memory.availableMB 1814.60546875! proc.memory.rssMB 3122.609375 cpu25.273494984791228 proc.cpu.threads17) disk./.usagePercent77.23797971920602 disk.dm-0.in 9007.83984375 disk.sda2.out0% disk.nvme0c0n1.out91698.52734375* proc.memory.percent19.058895111083984 network.recv 1.300102e+06/^(: ΁ gpu.0.gpu96.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51716613769531* gpu.0.memoryAllocatedBytes 40816934912 gpu.0.temp45.0 gpu.0.powerWatts218.508' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent43.7016 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0G(: ݡ gpu.0.gpu99.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp45.0 gpu.0.powerWatts154.941' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent30.9882 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 {Uw(:  disk.sda2.in0 disk.dm-0.in 9007.83984375# disk.dm-0.out91975.05126953125* proc.memory.percent19.058895111083984 cpu25.274070663852115 disk.drbd1260.out0.5625& disk.nvme0n1.out91975.05126953125$ memory_percent88.92450332641602' proc.memory.availableMB 1814.609375! proc.memory.rssMB 3122.609375 proc.cpu.threads17 network.recv 1.303894e+06) disk./.usagePercent77.26059747284108 disk.sda1.in0 disk.sda1.out0 disk.sda3.out 122.984375 disk.sda2.out0 disk.drbd1260.in0! disk.nvme0n1.in 9007.83984375 network.sent 5.271193e+06% disk./.usageGB1151.2387351989746 disk.sda3.in0 disk.sda.out 122.984375# disk.nvme0c0n1.in 9007.83984375( disk.nvme0c0n1.out91975.05126953125 disk.dm-1.in0 disk.dm-1.out 122.984375 disk.sda.in0-%2- _wandbruntime4008  _runtime4008 & lr-AdamW/pg20.0009170996961734626 trainer/global_step2299 % _timestamp1.7884691180955756e+09 _runtime4008.136235869 _step183 & lr-AdamW/pg10.0009170996961734626ͩ % _timestamp1.7884691180955756e+09 _runtime4008.136235869 _step183 & lr-AdamW/pg10.0009170996961734626 & lr-AdamW/pg20.0009170996961734626 trainer/global_step2299\?2- _wandbruntime4008  _runtime4008'O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated26472833 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated108736107223040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated101938744620544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated504040263120896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current63133696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed26472353 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated504040263120896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak881 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed101938715522560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated43110167 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5322402781184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed504037721329664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5319886417920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed10160594 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated498717860339712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated32920944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed108736014991360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated498717860339712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed498717834911744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10160597 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated16312236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated478466728954192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5319886417920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280ѝd O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current63133696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed26472353 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated504040263120896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak881 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 14008960q-ę00 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed101938715522560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated43110167 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5322402781184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed504037721329664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5319886417920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed10160594 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated498717860339712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated32920944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed108736014991360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated498717860339712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed498717834911744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10160597 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated16312236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated478466728954192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5319886417920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated26472833 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated108736107223040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated101938744620544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated504040263120896])2- _wandbruntime4010  _runtime4010Wi'M RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed101982776891904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed76043653 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed76043653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed10164926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61023232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5324678010880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _step185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current474 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated26484103 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed483984407697566 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated478669257663776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed498928825287168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed498928825287168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak881 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6800277324800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated498928850715136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1202 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6800216301568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed26483629 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed43108834 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated101982805989888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated43128544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed43108834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed108782993193472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3204448256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5322161634304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current471 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated504253528726016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed16318703 _runtime4010.130347276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10164929 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated43128544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90121216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed478669232235808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5322161634304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated76063369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3674b'N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 $ _timestamp1.788469120088175e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated26484103 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed483984407697566 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated478669257663776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed498928825287168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed498928825287168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak881 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6800277324800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated498928850715136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1202 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6800216301568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed26483629 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed43108834 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated101982805989888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated43128544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed43108834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed108782993193472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3204448256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5322161634304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current471 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated504253528726016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed16318703 _runtime4010.130347276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10164929 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated43128544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current90121216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed478669232235808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5322161634304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated76063369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3674 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed101982776891904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed76043653 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed76043653 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed10164926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current61023232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5324678010880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 _step185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current474 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed514701028748288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90134528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated27035176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated104141813087744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884692018181322e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5433675065344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed33614988 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated514703570539520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed111084256142336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10377197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6942533189120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed514701028748288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current90134528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated27035176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated104141813087744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 % _timestamp1.7884692018181322e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5433675065344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed33614988 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated514703570539520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed111084256142336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10377197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 6942533189120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated77678321 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed111131243531264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed488795667715008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated27046422 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed494224483091474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed77658605 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6945457098240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated33628875 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated44049446 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated16664893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed33628869 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed509478344058368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed514914294340096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated104185874457088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated509478369486336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated509478369486336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5435950281728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed27045970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated44049446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed16664444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 trainer/global_step2349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed509478344058368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated488795693142976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6945398172160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated494227022888062 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed10381526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1202 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 _step189 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5435950281728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated514916836144640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5438466658304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5438466658304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak881 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 % _timestamp1.7884692033999555e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3674 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5431329745086 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed44029736 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464AL'N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 % _timestamp1.7884692033999555e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc3674 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5431329745086 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed44029736 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak 451453440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated77678321 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed77658605 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated77678321 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed111131243531264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed488795667715008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated27046422 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current19710 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed494224483091474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed77658605 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 6945457098240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated33628875 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated44049446 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated16664893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed33628869 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed509478344058368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed514914294340096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated104185874457088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated509478369486336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated509478369486336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5435950281728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed27045970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated44049446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed16664444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 trainer/global_step2349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed509478344058368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated488795693142976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 6945398172160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated494227022888062 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed10381526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1202 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current88024064 _step189 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5435950281728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated514916836144640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5438466658304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5438466658304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current58926080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak881 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588\D2- _wandbruntime4093  _runtime4093 {>Z 2loss/SRqqt2/assignment_loss1.9571841835086026 2loss/SRqqt3/assignment_loss2.1473606632671496 1loss/FBt1/assignment_loss0.12988792529634524 .loss/FBt3/detection_loss3.056923873284284 epoch3 /loss/FRt4/assignment_loss3.240402349777389 1loss/FBt2/assignment_loss0.28728597573111375 1loss/FBt3/assignment_loss0.19207413411726626 1loss/SRqqt4/detection_loss1.7138270298319478 /loss/FBt1/detection_loss0.7260800993808821 1loss/SRqqt2/detection_loss2.5220739690225518 %loss/total_loss49.33738405563727 trainer/global_step2349 0loss/FBt4/assignment_loss0.2769047943601275 .loss/FRt3/detection_loss3.173994586936065 .loss/FBt2/detection_loss3.198851816356182 2loss/SRqqt1/assignment_loss1.1573954865038394 /loss/FRt1/detection_loss1.8882547015637423 .loss/FRt2/detection_loss2.814396623770396 1loss/SRqqt3/detection_loss2.5997964489287226 _step190 $ _timestamp1.788469203403547e+09 0loss/FRt1/assignment_loss1.7061503376580973 /loss/FRt4/detection_loss2.1045065339197193 /loss/FBt4/detection_loss3.0735050685265484 /loss/FRt2/assignment_loss4.371097064018249 0loss/FRt3/assignment_loss3.7997590846249034 0loss/SRqqt1/detection_loss1.115556148469448 2loss/SRqqt4/assignment_loss2.0841151567836924 _runtime4093.444450491"7Q_ /loss/FRt2/assignment_loss4.371097064018249 0loss/FRt3/assignment_loss3.7997590846249034 0loss/SRqqt1/detection_loss1.115556148469448 2loss/SRqqt4/assignment_loss2.0841151567836924 _runtime4093.444450491 2loss/SRqqt2/assignment_loss1.9571841835086026 2loss/SRqqt3/assignment_loss2.1473606632671496 1loss/FBt1/assignment_loss0.12988792529634524 .loss/FBt3/detection_loss3.056923873284284 epoch3 /loss/FRt4/assignment_loss3.240402349777389 1loss/FBt2/assignment_loss0.28728597573111375 1loss/FBt3/assignment_loss0.19207413411726626 1loss/SRqqt4/detection_loss1.7138270298319478 /loss/FBt1/detection_loss0.7260800993808821 1loss/SRqqt2/detection_loss2.5220739690225518 %loss/total_loss49.33738405563727 trainer/global_step2349 0loss/FBt4/assignment_loss0.2769047943601275 .loss/FRt3/detection_loss3.173994586936065 .loss/FBt2/detection_loss3.198851816356182 2loss/SRqqt1/assignment_loss1.1573954865038394 /loss/FRt1/detection_loss1.8882547015637423 .loss/FRt2/detection_loss2.814396623770396 1loss/SRqqt3/detection_loss2.5997964489287226 _step190 $ _timestamp1.788469203403547e+09 0loss/FRt1/assignment_loss1.7061503376580973 /loss/FRt4/detection_loss2.1045065339197193 /loss/FBt4/detection_loss3.0735050685265484HD (: Կ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp45.0 gpu.0.powerWatts231.349' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.2698 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0={(: Կ disk.dm-0.in 9383.83984375! proc.memory.rssMB 3122.609375 cpu25.276974131169688 disk.sda3.out 126.109375 network.sent 5.386459e+06 network.recv 1.32962e+06) disk./.usagePercent77.36122737835504 disk.sda1.in0 disk.sda1.out0) proc.memory.availableMB 1814.00390625* proc.memory.percent19.058895111083984! disk.nvme0n1.in 9383.83984375 disk.sda.in0 disk.sda.out 126.109375# disk.nvme0c0n1.in 9383.83984375( disk.nvme0c0n1.out93909.61279296875 disk.drbd1260.in0 disk.drbd1260.out 0.57421875 proc.cpu.threads17$ disk./.usageGB1152.738193511963# disk.dm-0.out93909.61279296875 disk.sda3.in0 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.dm-1.out 126.109375$ memory_percent88.92819881439209& disk.nvme0n1.out93909.61279296875+(: ҽ& disk.nvme0n1.out94161.10498046875 disk.sda1.in0 disk.sda3.out 126.61328125 disk.sda2.in0# disk.nvme0c0n1.in 9383.83984375 disk.dm-1.in0 disk.drbd1260.out 0.57421875$ memory_percent88.92667293548584 network.recv 1.335315e+06% disk./.usageGB1152.7381134033203 disk.sda.in0 disk.sda.out 126.61328125 disk.dm-1.out 126.61328125 network.sent 5.473498e+06) disk./.usagePercent77.36122200219683! disk.nvme0n1.in 9383.83984375 disk.dm-0.in 9383.83984375# disk.dm-0.out94161.10498046875 disk.sda2.out0* proc.memory.percent19.058895111083984 cpu25.277525674194074 disk.sda1.out0 disk.sda3.in0( disk.nvme0c0n1.out94161.10498046875 disk.drbd1260.in0) proc.memory.availableMB 1814.25390625! proc.memory.rssMB 3122.609375 proc.cpu.threads17Gp(:  gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts205.758' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent41.1516 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0(: ϖ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts236.713' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent47.342600000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors04~x(:  disk.drbd1260.in0 disk.drbd1260.out0.578125! disk.nvme0n1.in 9383.83984375# disk.dm-0.out94639.52685546875 disk.sda1.in0 disk.sda3.out 127.21875 disk.sda.in0 disk.sda2.in0 network.sent 5.477613e+06% disk./.usageGB1152.7146453857422 disk.dm-1.in0 disk.dm-0.in 9383.83984375 disk.sda3.in0( disk.nvme0c0n1.out94639.52685546875! proc.memory.rssMB 3122.609375 cpu25.278004537259147) disk./.usagePercent77.35964704384618 disk.dm-1.out 127.21875 disk.sda1.out0 disk.sda.out 127.21875) proc.memory.availableMB 1814.49609375* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.recv 1.337942e+06& disk.nvme0n1.out94639.52685546875 disk.sda2.out0# disk.nvme0c0n1.in 9383.83984375$ memory_percent88.92519474029541|(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp45.0 gpu.0.powerWatts221.459' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent44.2918 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0B9)(:  disk.sda2.in0 disk.dm-1.in0 disk.drbd1260.in0 disk.dm-0.in 9383.83984375 disk.sda3.in0 disk.sda2.out0( disk.nvme0c0n1.out94985.65185546875 disk.dm-1.out 127.59765625! proc.memory.rssMB 3122.609375 cpu25.278430781254897 network.sent 5.483755e+06! disk.nvme0n1.in 9383.83984375 disk.sda1.out0 disk.sda3.out 127.59765625# disk.nvme0c0n1.in 9383.83984375* proc.memory.percent19.058895111083984 proc.cpu.threads17 network.recv 1.342501e+06% disk./.usageGB1152.7380905151367 disk.sda.out 127.59765625 disk.drbd1260.out0.578125$ memory_percent88.92672061920166) proc.memory.availableMB 1814.24609375 disk.sda.in0) disk./.usagePercent77.36122046615162& disk.nvme0n1.out94985.65185546875# disk.dm-0.out94985.65185546875 disk.sda1.in0!P|(:  disk.sda2.in0! disk.nvme0c0n1.in 9490.640625 disk.dm-1.in0( proc.memory.availableMB 1814.2109375! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984 network.recv 1.346653e+06% disk./.usageGB1151.5407028198242# disk.dm-0.out95334.27197265625 disk.sda1.in0 disk.sda.out 127.98046875 cpu25.278885498505204) disk./.usagePercent77.28086277324401 disk.sda3.in0 disk.sda3.out 127.98046875 disk.sda.in0 disk.sda2.out0 proc.cpu.threads17 network.sent 5.489275e+06& disk.nvme0n1.out95334.27197265625 disk.sda1.out0( disk.nvme0c0n1.out95334.27197265625 disk.dm-1.out 127.98046875 disk.drbd1260.in0 disk.drbd1260.out 0.58203125$ memory_percent88.92693519592285 disk.nvme0n1.in 9490.640625 disk.dm-0.in 9490.640625RZ(:  gpu.0.gpu69.0 gpu.0.memory13+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts243.139' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent48.62780000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0^z(: ζ network.sent 5.494991e+06 disk.sda2.out0 disk.drbd1260.in0 disk.sda1.in0 disk.sda1.out0( proc.memory.availableMB 1814.4453125! proc.memory.rssMB 3122.609375* proc.memory.percent19.058895111083984! disk.nvme0c0n1.in 9490.640625 disk.nvme0n1.in 9490.640625& disk.nvme0n1.out95902.43994140625 disk.sda3.in0 disk.sda.in0 proc.cpu.threads17) disk./.usagePercent77.28679267575961 disk.sda2.in0( disk.nvme0c0n1.out95902.43994140625 disk.dm-1.out 128.35546875 disk.drbd1260.out 0.58203125 disk.dm-0.in 9490.640625 disk.sda3.out 128.35546875 disk.sda.out 128.35546875 network.recv 1.35109e+06$ disk./.usageGB1151.629062652588 disk.dm-1.in0# disk.dm-0.out95902.43994140625$ memory_percent88.92550468444824 cpu25.279208359528173o(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts228.429' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.6858 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0>J2- _wandbruntime4173  _runtime4173D _step191 & lr-AdamW/pg10.0008883114339998417 & lr-AdamW/pg20.0008883114339998417 trainer/global_step2399 % _timestamp1.7884692832306068e+09 _runtime4173.271179945"| & lr-AdamW/pg10.0008883114339998417 & lr-AdamW/pg20.0008883114339998417 trainer/global_step2399 % _timestamp1.7884692832306068e+09 _runtime4173.271179945 _step191IӨ2- _wandbruntime4173  _runtime4173Vq#&O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed34309038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed10593794 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed113432521085952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _runtime4173.274243793 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed525364336166912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed44932287 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated525366877958144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated34309044 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated519816897882112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5549980076032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7087727568384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10593797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated27597547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 trainer/global_step2399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5542697404732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5549980076032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed519816872454144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed525364336166912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed44932287 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated525366877958144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated34309044 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated519816897882112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5549980076032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7087727568384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated10593797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated27597547 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 trainer/global_step2399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5542697404732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5549980076032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed519816872454144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5552255305728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated44970348 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated525580143563264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated106388942924288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated504467098281970 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated79293273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated27608751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 7090546370048 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated520027888257536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5549738929152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5549738929152 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed17010160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 7090605296128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5542455291174 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 % _timestamp1.7884692847352705e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated520027888257536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated79293273 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated113479548220416 _step193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed79273557 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5552255305728 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed27608286 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated34322925 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated44970348 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5544969659794 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed34322919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed520027862829568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10598129ρ'N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated 5544969659794 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed34322919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed520027862829568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10598129 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5552255305728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated44970348 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated525580143563264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated106388942924288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated504467098281970 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated 40265318400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated79293273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated27608751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed 7090546370048 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated520027888257536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak 3142753280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed 5549738929152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5549738929152 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current 2514368620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed17010160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 7090605296128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5542455291174 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak 35901791232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 % _timestamp1.7884692847352705e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak2684 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated520027888257536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated79293273 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated113479548220416 _step193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed79273557 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5552255305728 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed27608286 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated34322925 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated44970348ؓj+2- _wandbruntime4174  _runtime41741%F[ 2loss/SRqqt1/assignment_loss1.1020667963596278 /loss/FRt1/detection_loss1.8976575216421714 /loss/FBt4/detection_loss2.6272931933403014 &loss/total_loss47.594096791573456 % _timestamp1.7884692847378752e+09 1loss/SRqqt3/assignment_loss2.010686322379468 /loss/FRt2/detection_loss2.3793355395097153 1loss/SRqqt3/detection_loss2.5036292227346504 /loss/FBt3/detection_loss2.8984472933022873 trainer/global_step2399 2loss/SRqqt4/assignment_loss1.8500749901071325 0loss/FBt2/assignment_loss0.5233671310823411 /loss/SRqqt2/detection_loss2.39626538402894 1loss/FBt3/assignment_loss0.24104012067444905 /loss/FRt4/detection_loss2.4415927010098684 1loss/SRqqt4/detection_loss1.8278013548335514 /loss/FBt2/detection_loss2.7070482581853867 0loss/FRt1/assignment_loss1.8357441065075633 .loss/FRt2/assignment_loss3.99099310785532 2loss/SRqqt2/assignment_loss2.1357777811148586 1loss/SRqqt1/detection_loss0.9991212207390818 _runtime4174.778191413 .loss/FRt3/assignment_loss3.70314114485214 epoch3 0loss/FRt4/assignment_loss3.5797137009728814 1loss/FBt1/assignment_loss0.12779806153314785 1loss/FBt4/assignment_loss0.22796776954140416 .loss/FRt3/detection_loss2.820859195581123 _step194 /loss/FBt1/detection_loss0.7666748736860456ϐ! _runtime4174.778191413 .loss/FRt3/assignment_loss3.70314114485214 epoch3 0loss/FRt4/assignment_loss3.5797137009728814 1loss/FBt1/assignment_loss0.12779806153314785 1loss/FBt4/assignment_loss0.22796776954140416 .loss/FRt3/detection_loss2.820859195581123 _step194 /loss/FBt1/detection_loss0.7666748736860456 2loss/SRqqt1/assignment_loss1.1020667963596278 /loss/FRt1/detection_loss1.8976575216421714 /loss/FBt4/detection_loss2.6272931933403014 &loss/total_loss47.5940967915()?73456 % _timestamp1.7884692847378752e+09 1loss/SRqqt3/assignment_loss2.010686322379468 /loss/FRt2/detection_loss2.3793355395097153 1loss/SRqqt3/detection_loss2.5036292227346504 /loss/FBt3/detection_loss2.8984472933022873 trainer/global_step2399 2loss/SRqqt4/assignment_loss1.8500749901071325 0loss/FBt2/assignment_loss0.5233671310823411 /loss/SRqqt2/detection_loss2.39626538402894 1loss/FBt3/assignment_loss0.24104012067444905 /loss/FRt4/detection_loss2.4415927010098684 1loss/SRqqt4/detection_loss1.8278013548335514 /loss/FBt2/detection_loss2.7070482581853867 0loss/FRt1/assignment_loss1.8357441065075633 .loss/FRt2/assignment_loss3.99099310785532 2loss/SRqqt2/assignment_loss2.1357777811148586 1loss/SRqqt1/detection_loss0.9991212207390818Ocv(: ! proc.memory.rssMB 3122.609375 cpu25.279898765002283 network.sent 5.500115e+06) disk./.usagePercent77.28685898171098 disk.nvme0n1.in 9490.640625 disk.dm-0.in 9490.640625 disk.sda1.out0 disk.sda.out 128.796875! disk.nvme0c0n1.in 9490.640625( disk.nvme0c0n1.out96057.58447265625# disk.dm-0.out96057.58447265625 disk.sda3.in0 disk.sda3.out 128.796875 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out 0.5859375) proc.memory.availableMB 1814.37890625* proc.memory.percent19.058895111083984 network.recv 1.354656e+06 disk.sda.in0 disk.dm-1.out 128.796875 proc.cpu.threads17% disk./.usageGB1151.6300506591797& disk.nvme0n1.out96057.58447265625 disk.sda1.in0 disk.sda2.in0 disk.sda2.out0$ memory_percent88.92590999603271 (: Ŷ gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts245.251' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.0502 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0xV]Q(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts215.864' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent43.1728 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Jw(:  disk.sda1.in0( disk.nvme0c0n1.out96294.11572265625 disk.dm-1.out 129.17578125* proc.memory.percent19.058895111083984 cpu25.280468864928526 disk.dm-0.in 9490.640625 disk.sda3.in0 disk.sda3.out 129.17578125 disk.sda.in0 disk.sda.out 129.17578125! disk.nvme0c0n1.in 9490.640625$ proc.memory.availableMB1814.625 network.sent 5.587456e+06 disk.sda1.out0 disk.drbd1260.in0! proc.memory.rssMB 3122.609375 proc.cpu.threads17% disk./.usageGB1151.6064529418945 disk.nvme0n1.in 9490.640625# disk.dm-0.out96294.11572265625 disk.sda2.in0 disk.sda2.out0 disk.dm-1.in0 disk.drbd1260.out 0.5859375$ memory_percent88.92440795898438 network.recv 1.360638e+06) disk./.usagePercent77.28527531910417& disk.nvme0n1.out96294.11572265625((: ࢉ gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp40.0 gpu.0.powerWatts115.416' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent23.083199999999998 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0N's(:  disk.sda3.in0 disk.sda.in0 disk.dm-1.out 129.859375& proc.memory.availableMB 1814.21875% disk./.usageGB1150.5402946472168# disk.dm-0.out96860.69482421875 disk.sda2.in0! disk.nvme0c0n1.in 9602.921875* proc.memory.percent19.058895111083984 cpu25.29683841587832 proc.cpu.threads17 disk.sda1.in0 disk.sda.out 129.859375( disk.nvme0c0n1.out96860.69482421875 disk.drbd1260.in0 disk.drbd1260.out 0.58984375$ memory_percent88.92688751220703! proc.memory.rssMB 3122.609375 network.recv 1.364704e+06& disk.nvme0n1.out96860.69482421875 disk.sda1.out0 disk.sda3.out 129.859375 disk.sda2.out0 disk.dm-1.in0 network.sent 5.594875e+06) disk./.usagePercent77.21372454138195 disk.nvme0n1.in 9602.921875 disk.dm-0.in 9602.9218752޻m}(: ) proc.memory.availableMB 1814.54296875 disk.dm-0.in 9602.921875! proc.memory.rssMB 3122.609375 cpu25.313016849198597 proc.cpu.threads17 network.recv 1.368842e+06) disk./.usagePercent77.21530999604151% disk./.usageGB1150.5639190673828 disk.sda3.in0 disk.sda3.out 130.37109375 disk.sda.in0 disk.sda.out 130.37109375 disk.sda2.out0 disk.sda1.in0 disk.sda1.out0 disk.sda2.in0! disk.nvme0c0n1.in 9602.921875( disk.nvme0c0n1.out97124.27685546875 disk.dm-1.out 130.37109375 disk.drbd1260.out 0.58984375* proc.memory.percent19.058895111083984 network.sent 5.598716e+06# disk.dm-0.out97124.27685546875 disk.dm-1.in0 disk.drbd1260.in0 disk.nvme0n1.in 9602.921875& disk.nvme0n1.out97124.27685546875$ memory_percent88.924908638000496NH(: ּ gpu.0.gpu98.0 gpu.0.memory22+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp45.0 gpu.0.powerWatts265.092' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent53.0184 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0_(:  gpu.0.gpu0.0 gpu.0.memory0+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp38.0 gpu.0.powerWatts114.882' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent22.9764 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Rv(: & disk.nvme0n1.out97317.05029296875 disk.sda1.in0 disk.sda1.out0 disk.dm-1.in0# disk./.usageGB1150.56396484375! disk.nvme0c0n1.in 9602.921875 disk.dm-0.in 9602.921875( proc.memory.availableMB 1814.4296875* proc.memory.percent19.058895111083984 cpu25.33563775713344 disk.sda2.out0( disk.nvme0c0n1.out97317.05029296875 disk.drbd1260.in0 disk.drbd1260.out0.59375# disk.dm-0.out97317.05029296875$ memory_percent88.92560005187988! proc.memory.rssMB 3122.609375 disk.dm-1.out 130.86328125 network.sent 5.605599e+06 network.recv 1.372097e+06 disk.sda3.in0 disk.sda3.out 130.86328125 disk.sda.out 130.86328125 disk.sda2.in0 proc.cpu.threads17) disk./.usagePercent77.21531306813192 disk.sda.in0 disk.nvme0n1.in 9602.921875`42- _wandbruntime4250  _runtime4250u*;b @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9612084031105042 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.22369182109832764 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 9"Purity/*FRt1SRqqt1FBt/event_purity0.0736161395907402 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 & Purity/2FRt2SRqqt1FBt/FBt_purity1 ( Purity/*FRt3SRqqt2FBt/FBt_purity0.5 ("Purity/4FRt1SRqqt*FBt/event_purity0 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.20665742456912994 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 & Purity/3FRt0SRqqt1FBt/FBt_purity1 jet/accuracy_5_of_60 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 particle/accuracy_2_of_21 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity0 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9855555295944214 7 Purity/2FRt1SRqqt2FBt/FRt_purity0.3179824650287628 ("Purity/*FRt0SRqqt4FBt/event_purity0 jet/accuracy_7_of_70 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9714804887771606 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.3571428656578064 , Purity/1FRt3SRqqt*FBt/FBt_purity0.96875 ;"Purity/2FRt*SRqqt0FBt/event_purity0.001240694778971374 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 ("Purity/3FRt0SRqqt2FBt/event_purity0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity0 ("Purity/2FRt2SRqqt2FBt/event_purity0 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ("Purity/2FRt3SRqqt0FBt/event_purity0 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9861111044883728 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 6 Purity/*FRt0SRqqt4FBt/FBt_purity0.699999988079071 & Purity/3FRt1SRqqt2FBt/FBt_purity1 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 particle/accuracy_5_of_61 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.39841270446777344 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 7"Purity/0FRt0SRqqt*FBt/event_purity0.81014484167099 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9822250604629517 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.5350877046585083 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 ("Purity/2FRt1SRqqt2FBt/event_purity0 ) Purity/1FRt*SRqqt4FBt/FBt_purity0.75 ("Purity/1FRt2SRqqt*FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 +jet/accuracy_2_of_20.08100736886262894 ) Purity/*FRt1SRqqt3FBt/FRt_purity0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.1797143965959549 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.7083333134651184 ("Purity/*FRt4SRqqt*FBt/event_purity0 :"Purity/0FRt*SRqqt*FBt/event_purity0.32967686653137207 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.31308239698410034 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity0 & Purity/4FRt2SRqqt*FBt/FRt_purity0 & Purity/4FRt2SRqqt0FBt/FRt_purity0 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2545413076877594 8 Purity/1FRt*SRqqt0FBt/FRt_purity0.21964110434055328 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.16021355986595154 & Purity/1FRt3SRqqt2FBt/FRt_purity1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity0.19120079278945923 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.17853040993213654 7 Purity/1FRt1SRqqt1FBt/FRt_purity0.3108481764793396 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ("Purity/3FRt3SRqqt*FBt/event_purity0 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.25110387802124023 7 Purity/*FRt*SRqqt0FBt/FRt_purity0.2141963541507721 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt3SRqqt0FBt/event_purity0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.5 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9929931163787842 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.2331511527299881 :"Purity/2FRt0SRqqt2FBt/event_purity0.01785714365541935 jet/accuracy_4_of_50 particle/accuracy_1_of_41 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.19924761354923248 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 :"Purity/0FRt0SRqqt3FBt/event_purity0.13333334028720856 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 7 Purity/2FRt*SRqqt*FBt/FRt_purity0.2479925900697708 8 Purity/2FRt*SRqqt0FBt/FRt_purity0.20610344409942627 + Purity/0FRt*SRqqt4FBt/FBt_purity0.6875 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.24298721551895142 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.17775459587574005 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ("Purity/2FRt2SRqqt1FBt/event_purity0 ,particle/sensitivity0.24017666280269623 & Purity/1FRt1SRqqt4FBt/FRt_purity0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.31315839290618896 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9886924624443054 ;"Purity/1FRt1SRqqt*FBt/event_purity0.011674611829221249 ("Purity/1FRt3SRqqt1FBt/event_purity0 ("Purity/3FRt*SRqqt1FBt/event_purity0 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 ("Purity/3FRt3SRqqt0FBt/event_purity0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.3857017159461975 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.22881494462490082 7 Purity/2FRt0SRqqt0FBt/FRt_purity0.1658840775489807 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.31739628314971924 & Purity/3FRt*SRqqt*FBt/FBt_purity1 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.8333333730697632 7 Purity/*FRt1SRqqt1FBt/FBt_purity0.9938223361968994 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 # _timestamp1.78846936008347e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9736158847808838 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.8822223544120789 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.22948196530342102 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.22213882207870483 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt2SRqqt0FBt/event_purity0 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt*FBt/FRt_purity0.2222222536802292 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.20694446563720703 ("Purity/3FRt2SRqqt0FBt/event_purity0 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 particle/accuracy_2_of_41 & Purity/3FRt2SRqqt1FBt/FBt_purity1 jet/accuracy_5_of_50 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.1785714328289032 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.2083333283662796 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9792627096176147 -"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.3125 particle/accuracy_4_of_51 jet/accuracy_2_of_70 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.21966226398944855 particle/accuracy_3_of_31 particle/accuracy_4_of_61 ) Purity/4FRt1SRqqt*FBt/FRt_purity0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity0.25 ("Purity/4FRt2SRqqt0FBt/event_purity0 :"Purity/0FRt*SRqqt3FBt/event_purity0.12345679849386215 9"Purity/1FRt0SRqqt3FBt/event_purity0.0714285746216774 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ("Purity/*FRt4SRqqt1FBt/event_purity0 & Purity/3FRt1SRqqt1FBt/FBt_purity1 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 +particle/specificity0.9482179284095764 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity1 :"Purity/*FRt0SRqqt3FBt/event_purity0.08749999850988388 :"Purity/0FRt0SRqqt2FBt/event_purity0.32632818818092346 :"Purity/1FRt0SRqqt2FBt/event_purity0.10900913923978806 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9873272180557251 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.28703704476356506 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.20370370149612427 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.20410628616809845 7 Purity/*FRt1SRqqt*FBt/FRt_purity0.2667010724544525 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.24521878361701965 9"Purity/1FRt0SRqqt0FBt/event_purity0.1830933839082718 particle/accuracy_6_of_61 * Purity/0FRt3SRqqt*FBt/FBt_purity0.875 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 +jet/accuracy_4_of_60.02631578966975212 particle/accuracy_6_of_71 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 particle/accuracy_4_of_41 :"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.32692307233810425 _runtime4250.126188169 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.23389437794685364 *jet/accuracy_1_of_60.9473684430122375 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.34633129835128784 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.1766233742237091 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 & Purity/*FRt4SRqqt*FBt/FBt_purity1 :"Purity/0FRt1SRqqt*FBt/event_purity0.16795898973941803 8 Purity/3FRt*SRqqt1FBt/FRt_purity0.25757575035095215 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 8 Purity/1FRt0SRqqt*FBt/FRt_purity0.24723216891288757 7 Purity/1FRt0SRqqt0FBt/FRt_purity0.1830933839082718 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 :"Purity/2FRt*SRqqt2FBt/event_purity0.01666666753590107 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.5175438523292542 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9941746592521667 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.21259558200836182 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 ("Purity/*FRt3SRqqt2FBt/event_purity0 <"Purity/*FRt2SRqqt0FBt/event_purity0.0050620995461940765 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 6 Purity/*FRt3SRqqt1FBt/FBt_purity0.976190447807312 :"Purity/1FRt*SRqqt1FBt/event_purity0.11610535532236099 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.25491753220558167 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.23881608247756958 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9930173754692078 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity1 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9810105562210083 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 :"Purity/1FRt*SRqqt*FBt/event_purity0.08760860562324524 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 ("Purity/2FRt1SRqqt0FBt/event_purity0 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.23539017140865326 7 Purity/*FRt2SRqqt0FBt/FRt_purity0.2194603532552719 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 ("Purity/1FRt3SRqqt*FBt/event_purity0 particle/accuracy_3_of_41 particle/accuracy_3_of_51 & Purity/3FRt0SRqqt2FBt/FBt_purity1 ("Purity/*FRt1SRqqt4FBt/event_purity0 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9957258701324463 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.18022897839546204 ;"Purity/0FRt2SRqqt0FBt/event_purity0.011537272483110428 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9847901463508606 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.38097161054611206 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 particle/accuracy_1_of_31 :"Purity/0FRt1SRqqt2FBt/event_purity0.05041005462408066 & Purity/2FRt*SRqqt1FBt/FBt_purity1 ;"Purity/2FRt0SRqqt*FBt/event_purity0.012312683276832104 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 ("Purity/0FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.3045594394207001 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 particle/accuracy_2_of_51 particle/accuracy_3_of_61 ( Purity/1FRt4SRqqt*FBt/FRt_purity0.5 ("Purity/1FRt4SRqqt0FBt/event_purity0 ( Purity/1FRt4SRqqt0FBt/FRt_purity0.5 ("Purity/3FRt4SRqqt*FBt/event_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9922876954078674 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 *jet/accuracy_1_of_40.7999711632728577 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.3201754093170166 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 ("Purity/1FRt1SRqqt2FBt/event_purity0 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 ("Purity/4FRt*SRqqt*FBt/event_purity0 * Purity/4FRt*SRqqt*FBt/FRt_purity0.125 * Purity/4FRt*SRqqt0FBt/FRt_purity0.125 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.1944444328546524 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 ;"Purity/0FRt2SRqqt*FBt/event_purity0.010748153552412987 *jet/accuracy_1_of_30.7469805479049683 *jet/accuracy_1_of_50.8943762183189392 ("Purity/*FRt2SRqqt3FBt/event_purity0 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 + Purity/0FRt0SRqqt4FBt/FBt_purity0.6875 ("Purity/1FRt*SRqqt4FBt/event_purity0 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 ;"Purity/2FRt0SRqqt1FBt/event_purity0.031481482088565826 jet/accuracy_1_of_70.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.2799282968044281 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 ;"Purity/1FRt1SRqqt1FBt/event_purity0.023209240287542343 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.2569444477558136 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.13333334028720856 ("Purity/0FRt3SRqqt1FBt/event_purity0 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9666666388511658 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.21489591896533966 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 +jet/accuracy_2_of_50.32190826535224915 ("Purity/0FRt*SRqqt4FBt/event_purity0 ("Purity/*FRt3SRqqt*FBt/event_purity0 6 Purity/1FRt0SRqqt2FBt/FRt_purity0.419704794883728 & Purity/3FRt0SRqqt*FBt/FBt_purity1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity 0.1796875 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.22345086932182312 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8580248355865479 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9764854907989502 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.2551843225955963 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/*FRt4SRqqt0FBt/event_purity0 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ;"Purity/1FRt1SRqqt0FBt/event_purity0.008930670097470284 +jet/accuracy_3_of_50.07391344755887985 _step195 :"Purity/*FRt1SRqqt0FBt/event_purity0.09970002621412277 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 & Purity/2FRt2SRqqt*FBt/FBt_purity1 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 particle/accuracy_1_of_11 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.39670467376708984 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.6428571343421936 ("Purity/*FRt1SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 9"Purity/*FRt0SRqqt1FBt/event_purity0.6644365787506104 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.20153924822807312 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 ("Purity/3FRt*SRqqt0FBt/event_purity0 particle/accuracy_1_of_21 ("Purity/2FRt4SRqqt0FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FRt_purity0.3461538553237915 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.38876041769981384 ;"Purity/2FRt*SRqqt1FBt/event_purity0.013346538878977299 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 *jet/accuracy_2_of_60.4035087525844574 & Purity/2FRt3SRqqt1FBt/FBt_purity1 particle/accuracy_7_of_71 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity0 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.2899305522441864 9"Purity/0FRt0SRqqt1FBt/event_purity0.9968518614768982 7 Purity/1FRt*SRqqt2FBt/FRt_purity0.4109748899936676 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.17086534202098846 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.14297236502170563 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.3880952298641205 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.22653190791606903 <"Purity/*FRt2SRqqt*FBt/event_purity0.0045618172734975815 :"Purity/1FRt0SRqqt1FBt/event_purity0.21464906632900238 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ("Purity/3FRt2SRqqt1FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FRt_purity1 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 8 Purity/2FRt3SRqqt*FBt/FRt_purity0.22999997437000275 5 Purity/2FRt3SRqqt0FBt/FRt_purity0.23549784719944 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ) Purity/1FRt1SRqqt3FBt/FRt_purity0.25 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.31290942430496216 9"Purity/1FRt0SRqqt*FBt/event_purity0.1836727112531662 7 Purity/3FRt0SRqqt1FBt/FRt_purity0.3333333134651184 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,jet/accuracy_4_of_40.001612903201021254 jet/accuracy_6_of_60 :"Purity/0FRt1SRqqt0FBt/event_purity0.18784432113170624 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.3165898025035858 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 7 Purity/2FRt*SRqqt2FBt/FRt_purity0.3927777409553528 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 (particle/accuracy0.8376176357269287 ("Purity/3FRt4SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt2FBt/event_purity0 5 Purity/*FRt0SRqqt2FBt/FRt_purity0.42391237616539 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 ("Purity/2FRt3SRqqt*FBt/event_purity0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.3333333432674408 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.39277777075767517 :"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.38461539149284363 particle/accuracy_5_of_51 ("Purity/0FRt3SRqqt0FBt/event_purity0 & Purity/0FRt4SRqqt*FBt/FBt_purity1 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.40714287757873535 ("Purity/0FRt4SRqqt*FBt/event_purity0 +jet/accuracy_3_of_60.07894736528396606 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 ("Purity/0FRt4SRqqt1FBt/event_purity0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_2_of_61 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.2652549147605896 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9808202385902405 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.19290071725845337 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt2SRqqt0FBt/event_purity0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 *jet/accuracy_1_of_20.6211100816726685 particle/accuracy_2_of_31 ("Purity/2FRt0SRqqt3FBt/event_purity0 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.34419333934783936 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 & Purity/3FRt*SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9800763726234436 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9949740171432495 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 ( Purity/1FRt3SRqqt2FBt/FBt_purity0.5 ("Purity/*FRt3SRqqt1FBt/event_purity0 :"Purity/1FRt*SRqqt0FBt/event_purity0.08004479110240936 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9945031404495239 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 ("Purity/3FRt*SRqqt*FBt/event_purity0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.20501701533794403 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.25 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9968518614768982 ;"Purity/0FRt2SRqqt1FBt/event_purity0.004608294926583767 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.35821667313575745 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.2037242203950882 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8492063879966736 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.19731684029102325 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.23298396170139313 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 *jet/accuracy_3_of_30.0177792776376009 epoch3 7 Purity/*FRt3SRqqt*FBt/FBt_purity0.9545454382896423 7 Purity/2FRt1SRqqt*FBt/FRt_purity0.2580519914627075 7 Purity/2FRt1SRqqt0FBt/FRt_purity0.2236194908618927 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 ("Purity/1FRt0SRqqt4FBt/event_purity0 6 Purity/3FRt3SRqqt*FBt/FRt_purity0.261904776096344 6 Purity/1FRt*SRqqt*FBt/FBt_purity0.979358971118927 ("Purity/3FRt1SRqqt*FBt/event_purity0 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity 0.1953125 6 Purity/3FRt3SRqqt0FBt/FRt_purity0.261904776096344 ("Purity/0FRt2SRqqt3FBt/event_purity0 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 +jet/accuracy_1_of_10.34701773524284363 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/*FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9815490245819092 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2926546633243561 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.31709179282188416 & Purity/1FRt0SRqqt4FBt/FRt_purity0 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 7 Purity/*FRt1SRqqt1FBt/FRt_purity0.3222063481807709 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.22674252092838287 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 & Purity/*FRt4SRqqt1FBt/FBt_purity1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.18489931523799896 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 7 Purity/2FRt4SRqqt*FBt/FRt_purity0.4000000059604645 7 Purity/2FRt4SRqqt0FBt/FRt_purity0.4000000I0059604645 & Purity/4FRt0SRqqt*FBt/FRt_purity0 & Purity/4FRt0SRqqt0FBt/FRt_purity0 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 ;"Purity/*FRt1SRqqt2FBt/event_purity0.028700603172183037 trainer/global_step2415 :"Purity/*FRt0SRqqt2FBt/event_purity0.23345503211021423 6 Purity/0FRt1SRqqt2FBt/FBt_purity0.962314784526825 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9047618508338928 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 particle/accuracy_1_of_51 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.2777777910232544 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.6333333253860474 'particle/f_score0.3159714937210083 ("Purity/1FRt2SRqqt2FBt/event_purity0 * Purity/0FRt3SRqqt1FBt/FBt_purity0.875 ("Purity/4FRt0SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.34951990842819214 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9927419424057007 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.22239290177822113 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.1968783289194107 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/2FRt3SRqqt1FBt/event_purity0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2083333432674408 ("Purity/1FRt2SRqqt1FBt/event_purity0 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt1SRqqt*FBt/event_purity0 & Purity/3FRt1SRqqt*FBt/FBt_purity1 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity0 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 8"Purity/*FRt1SRqqt*FBt/event_purity0.090814009308815 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 -"Purity/1FRt*SRqqt3FBt/event_purity0.0625 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9860703945159912 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 particle/accuracy_3_of_71 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity0.6238994598388672 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.20127779245376587 6 Purity/1FRt2SRqqt1FBt/FRt_purity0.243778795003891 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 ("Purity/2FRt*SRqqt3FBt/event_purity0 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8651341795921326 8 Purity/3FRt1SRqqt1FBt/FRt_purity0.23076923191547394 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt0SRqqt*FBt/event_purity0 :"Purity/1FRt*SRqqt2FBt/event_purity0.06752701103687286 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.2422444075345993 7 Purity/2FRt0SRqqt1FBt/FRt_purity0.3164682388305664 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 :"Purity/0FRt*SRqqt0FBt/event_purity0.14738963544368744 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9685100317001343 , Purity/1FRt*SRqqt3FBt/FBt_purity0.90625 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 ("Purity/3FRt0SRqqt1FBt/event_purity0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.3750000298023224 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 +jet/accuracy_3_of_40.03944425284862518 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.3750000298023224 ("Purity/0FRt3SRqqt*FBt/event_purity0 & Purity/3FRt2SRqqt*FBt/FBt_purity1 ("Purity/1FRt3SRqqt2FBt/event_purity0 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.29449525475502014 particle/accuracy_1_of_71 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.261904776096344 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.8771826028823853 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 & Purity/3FRt*SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.6666666865348816 6 Purity/*FRt1SRqqt*FBt/FBt_purity0.984598696231842 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 :"Purity/2FRt*SRqqt*FBt/event_purity0.00453285314142704 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.15763889253139496 particle/accuracy_2_of_71 & Purity/2FRt*SRqqt3FBt/FBt_purity1 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.33684012293815613 6 Purity/*FRt2SRqqt*FBt/FBt_purity0.984845757484436 ;"Purity/*FRt2SRqqt1FBt/event_purity0.001792114693671465 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.18784432113170624 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 ("Purity/1FRt4SRqqt*FBt/event_purity0 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 :"Purity/0FRt1SRqqt1FBt/event_purity0.12495119869709015 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.33061540126800537 ("Purity/2FRt2SRqqt*FBt/event_purity0 +jet/accuracy_2_of_40.26650553941726685 & Purity/*FRt0SRqqt4FBt/FRt_purity0 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 7 Purity/*FRt3SRqqt1FBt/FRt_purity0.2944444417953491 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 6 Purity/1FRt1SRqqt*FBt/FRt_purity0.272990882396698 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.25753429532051086 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity0 & Purity/2FRt0SRqqt3FBt/FBt_purity1 >&Purity/*FRt0SRqqt1FBt/event_proportion0.14591734111309052 ("Purity/3FRt1SRqqt1FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 :"Purity/0FRt*SRqqt2FBt/event_purity0.25910335779190063 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9760212302207947 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 & Purity/*FRt*SRqqt4FBt/FRt_purity0 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 7 Purity/2FRt*SRqqt1FBt/FRt_purity0.3648886978626251 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 7 Purity/*FRt0SRqqt1FBt/FRt_purity0.3519068658351898 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 & Purity/1FRt*SRqqt4FBt/FRt_purity0 9"Purity/*FRt0SRqqt0FBt/event_purity0.1530666947364807 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9673060774803162 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.2797619104385376 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 9"Purity/*FRt0SRqqt*FBt/event_purity0.3726472556591034 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 *loss/val_total_loss46.220626354312046 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_6_of_70 :"Purity/*FRt*SRqqt1FBt/event_purity0.38478487730026245 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9939448833465576 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.17777776718139648 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.1763174682855606 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.5333333611488342 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.2040543556213379 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 7 Purity/0FRt2SRqqt2FBt/FBt_purity0.9807692170143127 ) Purity/1FRt0SRqqt4FBt/FBt_purity0.75 ("Purity/1FRt1SRqqt4FBt/event_purity0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 ;"Purity/2FRt0SRqqt0FBt/event_purity0.003225806402042508 ("Purity/3FRt0SRqqt*FBt/event_purity0 ("Purity/3FRt2SRqqt*FBt/event_purity0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.24094201624393463 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ("Purity/0FRt4SRqqt0FBt/event_purity0 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 jet/accuracy_4_of_70 ("Purity/3FRt*SRqqt2FBt/event_purity0 9"Purity/*FRt*SRqqt*FBt/event_purity0.1813197284936905 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.21792234480381012 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 & Purity/0FRt4SRqqt1FBt/FBt_purity1 particle/accuracy_1_of_61 particle/accuracy_5_of_71 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.35555556416511536 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.25309279561042786 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.22267256677150726 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 jet/accuracy_5_of_70 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity1 7 Purity/1FRt1SRqqt2FBt/FRt_purity0.3263605237007141 ("Purity/3FRt0SRqqt0FBt/event_purity0 *jet/accuracy_2_of_30.1745252013206482 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.261904776096344 :"Purity/*FRt*SRqqt0FBt/event_purity0.09324809163808823 :"Purity/*FRt*SRqqt3FBt/event_purity0.08448275923728943 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.36666667461395264 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 6 Purity/1FRt0SRqqt2FBt/FBt_purity0.958686888217926 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 6 Purity/0FRt1SRqqt3FBt/FBt_purity0.800000011920929 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 :"Purity/*FRt*SRqqt2FBt/event_purity0.17645971477031708 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.1798771321773529 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.38235294818878174 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 ("Purity/2FRt1SRqqt1FBt/event_purity0 ("Purity/4FRt1SRqqt0FBt/event_purity0 ("Purity/4FRt2SRqqt*FBt/event_purity0 ("Purity/0FRt0SRqqt0FBt/event_purity1 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity0.15625 ) Purity/2FRt3SRqqt1FBt/FRt_purity0.25 jet/accuracy_3_of_70 particle/accuracy_4_of_71 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 * Purity/1FRt*SRqqt3FBt/FRt_purity0.375 8 Purity/3FRt0SRqqt*FBt/FRt_purity0.21969695389270782 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.11437907814979553 ("Purity/2FRt4SRqqt*FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9824560880661011 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9703595042228699 ("Purity/*FRt3SRqqt0FBt/event_purity0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.218288853764534 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.21220026910305023 ("Purity/0FRt2SRqqt2FBt/event_purity0 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 7validation_average_jet_accuracy0.33618563413619995 & Purity/2FRt3SRqqt*FBt/FBt_purity1/O :"Purity/1FRt*SRqqt1FBt/event_purity0.11610535532236099 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity0.25491753220558167 @&Purity/3FRt3SRqqt*FBt/event_proportion0.0005670362734235823 7 Purity/3FRt4SRqqt*FBt/FRt_purity0.1666666716337204 7 Purity/3FRt4SRqqt0FBt/FRt_purity0.1666666716337204 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity0.23881608247756958 ?&Purity/0FRt2SRqqt1FBt/event_proportion0.009261592291295528 7 Purity/1FRt*SRqqt1FBt/FBt_purity0.9930173754692078 @&Purity/4FRt1SRqqt*FBt/event_proportion6.300403038039804e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity1 >&Purity/*FRt1SRqqt0FBt/event_proportion0.33039313554763794 7 Purity/0FRt0SRqqt*FBt/FBt_purity0.9810105562210083 =&Purity/0FRt1SRqqt*FBt/event_proportion0.2305317521095276 ,&Purity/0FRt4SRqqt2FBt/event_proportion0 :"Purity/1FRt*SRqqt*FBt/event_purity0.08760860562324524 ?&Purity/2FRt0SRqqt1FBt/event_proportion0.008946572430431843 ("Purity/2FRt1SRqqt0FBt/event_purity0 >&Purity/2FRt1SRqqt0FBt/event_proportion0.03181703761219978 8 Purity/*FRt2SRqqt*FBt/FRt_purity0.23539017140865326 7 Purity/*FRt2SRqqt0FBt/FRt_purity0.2194603532552719 ,&Purity/*FRt3SRqqt3FBt/event_proportion0 ("Purity/1FRt3SRqqt*FBt/event_purity0 particle/accuracy_3_of_41 particle/accuracy_3_of_51 & Purity/3FRt0SRqqt2FBt/FBt_purity1 ("Purity/*FRt1SRqqt4FBt/event_purity0 7 Purity/*FRt0SRqqt1FBt/FBt_purity0.9957258701324463 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity0.18022897839546204 ;"Purity/0FRt2SRqqt0FBt/event_purity0.011537272483110428 7 Purity/1FRt1SRqqt*FBt/FBt_purity0.9847901463508606 8 Purity/2FRt1SRqqt1FBt/FRt_purity0.38097161054611206 ,&Purity/2FRt4SRqqt2FBt/event_proportion0 particle/accuracy_1_of_31 :"Purity/0FRt1SRqqt2FBt/event_purity0.05041005462408066 & Purity/2FRt*SRqqt1FBt/FBt_purity1 ;"Purity/2FRt0SRqqt*FBt/event_purity0.012312683276832104 ,&Purity/2FRt3SRqqt3FBt/event_proportion0 @&Purity/4FRt*SRqqt*FBt/event_proportion0.0001890120911411941 ("Purity/0FRt1SRqqt3FBt/event_purity0 >&Purity/*FRt*SRqqt1FBt/event_proportion0.27079132199287415 7 Purity/*FRt1SRqqt2FBt/FRt_purity0.3045594394207001 ,&Purity/2FRt3SRqqt4FBt/event_proportion0 particle/accuracy_2_of_51 particle/accuracy_3_of_61 ( Purity/1FRt4SRqqt*FBt/FRt_purity0.5 ("Purity/1FRt4SRqqt0FBt/event_purity0 ( Purity/1FRt4SRqqt0FBt/FRt_purity0.5 ("Purity/3FRt4SRqqt*FBt/event_purity0 ("Purity/1FRt1SRqqt3FBt/event_purity0 7 Purity/0FRt1SRqqt1FBt/FBt_purity0.9922876954078674 ,&Purity/2FRt4SRqqt3FBt/event_proportion0 ,&Purity/4FRt1SRqqt4FBt/event_proportion0 @&Purity/4FRt2SRqqt*FBt/event_proportion6.300403038039804e-05 *jet/accuracy_1_of_40.7999711632728577 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity0.3201754093170166 ?&Purity/0FRt0SRqqt3FBt/event_proportion0.002583165420219302 ("Purity/1FRt1SRqqt2FBt/event_purity0 ,&Purity/1FRt3SRqqt3FBt/event_proportion0 ?&Purity/2FRt1SRqqt1FBt/event_proportion0.007875503972172737 ?&Purity/2FRt2SRqqt0FBt/event_proportion0.015435988083481789 ("Purity/4FRt*SRqqt*FBt/event_purity0 * Purity/4FRt*SRqqt*FBt/FRt_purity0.125 * Purity/4FRt*SRqqt0FBt/FRt_purity0.125 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity0.1944444328546524 >&Purity/0FRt0SRqqt0FBt/event_proportion0.04082661122083664 ;"Purity/0FRt2SRqqt*FBt/event_purity0.010748153552412987 *jet/accuracy_1_of_30.7469805479049683 *jet/accuracy_1_of_50.8943762183189392 ("Purity/*FRt2SRqqt3FBt/event_purity0 @&Purity/1FRt0SRqqt4FBt/event_proportion6.300403038039804e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion0 + Purity/0FRt0SRqqt4FBt/FBt_purity0.6875 ("Purity/1FRt*SRqqt4FBt/event_purity0 =&Purity/1FRt1SRqqt0FBt/event_proportion0.1265120953321457 ;"Purity/2FRt0SRqqt1FBt/event_purity0.031481482088565826 jet/accuracy_1_of_70.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity0.2799282968044281 A&Purity/3FRt4SRqqt*FBt/event_proportion0.00012600806076079607 ) Purity/1FRt1SRqqt4FBt/FBt_purity0.75 ;"Purity/1FRt1SRqqt1FBt/event_purity0.023209240287542343 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity0.2569444477558136 :"Purity/2FRt3SRqqt1FBt/SRqqt_purity0.13333334028720856 ("Purity/0FRt3SRqqt1FBt/event_purity0 >&Purity/0FRt*SRqqt1FBt/event_proportion0.14787046611309052 7 Purity/1FRt1SRqqt2FBt/FBt_purity0.9666666388511658 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity0.21489591896533966 & Purity/1FRt3SRqqt1FBt/FBt_purity1 ,&Purity/3FRt4SRqqt4FBt/event_proportion0 +jet/accuracy_2_of_50.32190826535224915 ("Purity/0FRt*SRqqt4FBt/event_purity0 ("Purity/*FRt3SRqqt*FBt/event_purity0 6 Purity/1FRt0SRqqt2FBt/FRt_purity0.419704794883728 & Purity/3FRt0SRqqt*FBt/FBt_purity1 0"Purity/*FRt4SRqqt*FBt/SRqqt_purity 0.1796875 @&Purity/*FRt1SRqqt3FBt/event_proportion0.0006930443341843784 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity0.22345086932182312 7 Purity/0FRt*SRqqt3FBt/FBt_purity0.8580248355865479 7 Purity/0FRt0SRqqt2FBt/FBt_purity0.9764854907989502 >&Purity/1FRt*SRqqt1FBt/event_proportion0.09929435700178146 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity0.2551843225955963 @&Purity/2FRt2SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/*FRt4SRqqt0FBt/event_purity0 ?&Purity/0FRt1SRqqt2FBt/event_proportion0.008757560513913631 ?&Purity/1FRt0SRqqt2FBt/event_proportion0.015120967291295528 ;"Purity/1FRt1SRqqt0FBt/event_purity0.008930670097470284 +jet/accuracy_3_of_50.07391344755887985 _step195 :"Purity/*FRt1SRqqt0FBt/event_purity0.09970002621412277 ,&Purity/*FRt3SRqqt4FBt/event_proportion0 & Purity/2FRt2SRqqt*FBt/FBt_purity1 ,&Purity/4FRt4SRqqt1FBt/event_proportion0 particle/accuracy_1_of_11 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity0.39670467376708984 @&Purity/3FRt3SRqqt0FBt/event_proportion0.0005670362734235823 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity0.6428571343421936 ("Purity/*FRt1SRqqt3FBt/event_purity0 ?&Purity/*FRt*SRqqt3FBt/event_proportion0.004473286215215921 9"Purity/*FRt0SRqqt1FBt/event_purity0.6644365787506104 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity0.20153924822807312 >&Purity/1FRt1SRqqt1FBt/event_proportion0.03944052383303642 ("Purity/3FRt*SRqqt0FBt/event_purity0 particle/accuracy_1_of_21 ("Purity/2FRt4SRqqt0FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FRt_purity0.3461538553237915 8 Purity/*FRt*SRqqt2FBt/FRt_purity0.38876041769981384 ;"Purity/2FRt*SRqqt1FBt/event_purity0.013346538878977299 @&Purity/2FRt0SRqqt3FBt/event_proportion6.300403038039804e-05 *jet/accuracy_2_of_60.4035087525844574 & Purity/2FRt3SRqqt1FBt/FBt_purity1 particle/accuracy_7_of_71 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity0 ?&Purity/*FRt1SRqqt2FBt/event_proportion0.017011089250445366 @&Purity/*FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion0.0003150201519019902 ?&Purity/3FRt1SRqqt*FBt/event_proportion0.003717237850651145 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity0.2899305522441864 9"Purity/0FRt0SRqqt1FBt/event_purity0.9968518614768982 7 Purity/1FRt*SRqqt2FBt/FRt_purity0.4109748899936676 8 Purity/1FRt3SRqqt*FBt/FRt_purity0.17086534202098846 8 Purity/1FRt3SRqqt0FBt/FRt_purity0.14297236502170563 ?&Purity/2FRt2SRqqt1FBt/event_proportion0.003969253972172737 7 Purity/2FRt2SRqqt1FBt/FRt_purity0.3880952298641205 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity0.22653190791606903 <"Purity/*FRt2SRqqt*FBt/event_purity0.0045618172734975815 :"Purity/1FRt0SRqqt1FBt/event_purity0.21464906632900238 @&Purity/3FRt0SRqqt1FBt/event_proportion0.0011340725468471646 ("Purity/3FRt2SRqqt1FBt/event_purity0 & Purity/*FRt3SRqqt2FBt/FRt_purity1 ,&Purity/0FRt3SRqqt4FBt/event_proportion0 8 Purity/2FRt3SRqqt*FBt/FRt_purity0.22999997437000275 5 Purity/2FRt3SRqqt0FBt/FRt_purity0.23549784719944 ,&Purity/4FRt4SRqqt2FBt/event_proportion0 ) Purity/1FRt1SRqqt3FBt/FRt_purity0.25 >&Purity/*FRt*SRqqt2FBt/event_proportion0.06747731566429138 ,&Purity/*FRt4SRqqt3FBt/event_proportion0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity0.31290942430496216 9"Purity/1FRt0SRqqt*FBt/event_purity0.1836727112531662 7 Purity/3FRt0SRqqt1FBt/FRt_purity0.3333333134651184 ,&Purity/3FRt3SRqqt1FBt/event_proportion0 ,jet/accuracy_4_of_40.001612903201021254 jet/accuracy_6_of_60 :"Purity/0FRt1SRqqt0FBt/event_purity0.18784432113170624 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity0.3165898025035858 >&Purity/1FRt2SRqqt0FBt/event_proportion0.04731602966785431 7 Purity/2FRt*SRqqt2FBt/FRt_purity0.3927777409553528 7 Purity/2FRt1SRqqt2FBt/FBt_purity0.9868420958518982 (particle/accuracy0.8376176357269287 ("Purity/3FRt4SRqqt0FBt/event_purity0 ("Purity/3FRt1SRqqt2FBt/event_purity0 5 Purity/*FRt0SRqqt2FBt/FRt_purity0.42391237616539 ?&Purity/*FRt2SRqqt2FBt/event_proportion0.002079133177176118 >&Purity/2FRt1SRqqt*FBt/event_proportion0.04158266261219978 ("Purity/2FRt3SRqqt*FBt/event_purity0 ,&Purity/4FRt1SRqqt3FBt/event_proportion0 =&Purity/*FRt*SRqqt0FBt/event_proportion0.6160534024238586 ,&Purity/3FRt1SRqqt3FBt/event_proportion0 7 Purity/1FRt2SRqqt2FBt/FRt_purity0.3333333432674408 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity0.39277777075767517 :"Purity/3FRt1SRqqt1FBt/SRqqt_purity0.38461539149284363 particle/accuracy_5_of_51 ("Purity/0FRt3SRqqt0FBt/event_purity0 & Purity/0FRt4SRqqt*FBt/FBt_purity1 7 Purity/3FRt1SRqqt2FBt/FRt_purity0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity0 =&Purity/1FRt*SRqqt0FBt/event_proportion0.3135710656642914 8 Purity/2FRt0SRqqt2FBt/FRt_purity0.40714287757873535 ("Purity/0FRt4SRqqt*FBt/event_purity0 +jet/accuracy_3_of_60.07894736528396606 ?&Purity/*FRt3SRqqt*FBt/event_proportion0.020476309582591057 @&Purity/3FRt*SRqqt0FBt/event_proportion0.0071824598126113415 ("Purity/0FRt4SRqqt1FBt/event_purity0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity0.25 particle/accuracy_2_of_61 ,&Purity/4FRt2SRqqt2FBt/event_proportion0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity0.2652549147605896 7 Purity/0FRt*SRqqt*FBt/FBt_purity0.9808202385902405 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity0.19290071725845337 =&Purity/0FRt1SRqqt0FBt/event_proportion0.1694808453321457 @&Purity/0FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt2SRqqt0FBt/event_purity0 ,&Purity/3FRt3SRqqt2FBt/event_proportion0 ,&Purity/4FRt2SRqqt1FBt/event_proportion0 ,&Purity/4FRt3SRqqt0FBt/event_proportion0 *jet/accuracy_1_of_20.6211100816726685 particle/accuracy_2_of_31 ("Purity/2FRt0SRqqt3FBt/event_purity0 ?&Purity/0FRt0SRqqt2FBt/event_proportion0.029674898833036423 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity0.34419333934783936 A&Purity/3FRt1SRqqt2FBt/event_proportion0.00025201612152159214 & Purity/3FRt*SRqqt2FBt/FBt_purity1 7 Purity/*FRt0SRqqt*FBt/FBt_purity0.9800763726234436 7 Purity/1FRt1SRqqt1FBt/FBt_purity0.9949740171432495 @&Purity/2FRt4SRqqt0FBt/event_proportion0.0003150201519019902 ,&Purity/3FRt4SRqqt3FBt/event_proportion0 ( Purity/1FRt3SRqqt2FBt/FBt_purity0.5 ("Purity/*FRt3SRqqt1FBt/event_purity0 :"Purity/1FRt*SRqqt0FBt/event_purity0.08004479110240936 7 Purity/2FRt*SRqqt*FBt/FBt_purity0.9945031404495239 ,&Purity/2FRt0SRqqt4FBt/event_proportion0 ("Purity/3FRt*SRqqt*FBt/event_purity0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity0.20501701533794403 @&Purity/3FRt2SRqqt*FBt/event_proportion0.0028351815417408943 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity0.25 7 Purity/0FRt0SRqqt1FBt/FBt_purity0.9968518614768982 ;"Purity/0FRt2SRqqt1FBt/event_purity0.004608294926583767 @&Purity/1FRt*SRqqt3FBt/event_proportion0.0013230846961960196 8 Purity/1FRt0SRqqt1FBt/FRt_purity0.35821667313575745 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity0.2037242203950882 ,&Purity/1FRt4SRqqt2FBt/event_proportion0 7 Purity/*FRt1SRqqt3FBt/FBt_purity0.8492063879966736 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity0.19731684029102325 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity0.23298396170139313 ,&Purity/1FRt4SRqqt3FBt/event_proportion0 *jet/accuracy_3_of_30.0177792776376009 epoch3 7 Purity/*FRt3SRqqt*FBt/FBt_purity0.9545454382896423 7 Purity/2FRt1SRqqt*FBt/FRt_purity0.2580519914627075 7 Purity/2FRt1SRqqt0FBt/FRt_purity0.2236194908618927 ,&Purity/4FRt0SRqqt2FBt/event_proportion0 ,&Purity/4FRt1SRqqt1FBt/event_proportion0 ,&Purity/4FRt4SRqqt0FBt/event_proportion0 ("Purity/1FRt0SRqqt4FBt/event_purity0 6 Purity/3FRt3SRqqt*FBt/FRt_purity0.261904776096344 6 Purity/1FRt*SRqqt*FBt/FBt_purity0.979358971118927 ("Purity/3FRt1SRqqt*FBt/event_purity0 0"Purity/*FRt4SRqqt0FBt/SRqqt_purity 0.1953125 6 Purity/3FRt3SRqqt0FBt/FRt_purity0.261904776096344 ("Purity/0FRt2SRqqt3FBt/event_purity0 7 Purity/0FRt2SRqqt3FBt/FBt_purity0.6666666865348816 A&Purity/0FRt1SRqqt3FBt/event_proportion0.00044102821266278625 A&Purity/3FRt4SRqqt0FBt/event_proportion0.00012600806076079607 +jet/accuracy_1_of_10.34701773524284363 ("Purity/*FRt*SRqqt4FBt/event_purity0 ("Purity/*FRt2SRqqt2FBt/event_purity0 7 Purity/*FRt*SRqqt*FBt/FBt_purity0.9815490245819092 7 Purity/*FRt2SRqqt1FBt/FRt_purity0.2926546633243561 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity0.31709179282188416 & Purity/1FRt0SRqqt4FBt/FRt_purity0 =&Purity/*FRt0SRqqt2FBt/event_proportion0.0483240932226181 7 Purity/*FRt1SRqqt1FBt/FRt_purity0.3222063481807709 A&Purity/0FRt0SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity0.22674252092838287 @&Purity/3FRt0SRqqt2FBt/event_proportion6.300403038039804e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion0 & Purity/*FRt4SRqqt1FBt/FBt_purity1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity0.18489931523799896 ,&Purity/3FRt2SRqqt4FBt/event_proportion0 7 Purity/2FRt4SRqqt*FBt/FRt_purity0.4000000059604645 7 Purity/2FRt4SRqqt0FBt/FRt_purity0.4000000059604645 & Purity/4FRt0SRqqt*FBt/FRt_purity0 & Purity/4FRt0SRqqt0FBt/FRt_purity0 =&Purity/*FRt*SRqqt*FBt/event_proportion0.9591733813285828 ;"Purity/*FRt1SRqqt2FBt/event_purity0.028700603172183037 trainer/global_step2415 :"Purity/*FRt0SRqqt2FBt/event_purity0.23345503211021423 6 Purity/0FRt1SRqqt2FBt/FBt_purity0.962314784526825 @&Purity/1FRt0SRqqt3FBt/event_proportion0.0010710685746744275 7 Purity/1FRt0SRqqt3FBt/FBt_purity0.9047618508338928 ?&Purity/2FRt3SRqqt0FBt/event_proportion0.003654233878478408 particle/accuracy_1_of_51 7 Purity/3FRt2SRqqt1FBt/FRt_purity0.2777777910232544 9"Purity/0FRt*SRqqt3FBt/SRqqt_purity0.6333333253860474 'particle/f_score0.3159714937210083 ("Purity/1FRt2SRqqt2FBt/event_purity0 * Purity/0FRt3SRqqt1FBt/FBt_purity0.875 ("Purity/4FRt0SRqqt0FBt/event_purity0 ,&Purity/*FRt4SRqqt4FBt/event_proportion0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity0.34951990842819214 7 Purity/2FRt0SRqqt*FBt/FBt_purity0.9927419424057007 8 Purity/3FRt*SRqqt*FBt/FRt_purity0.22239290177822113 7 Purity/3FRt*SRqqt0FBt/FRt_purity0.1968783289194107 ("Purity/0FRt0SRqqt4FBt/event_purity0 ("Purity/2FRt3SRqqt1FBt/event_purity0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity0.2083333432674408 ("Purity/1FRt2SRqqt1FBt/event_purity0 @&Purity/1FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt1SRqqt*FBt/event_purity0 & Purity/3FRt1SRqqt*FBt/FBt_purity1 @&Purity/3FRt1SRqqt1FBt/event_proportion0.0009450604557059705 ,&Purity/3FRt4SRqqt2FBt/event_proportion0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity0 =&Purity/*FRt1SRqqt*FBt/event_proportion0.4482736885547638 8"Purity/*FRt1SRqqt*FBt/event_purity0.090814009308815 ?&Purity/0FRt3SRqqt*FBt/event_proportion0.006741431541740894 -"Purity/1FRt*SRqqt3FBt/event_purity0.0625 7 Purity/1FRt2SRqqt*FBt/FBt_purity0.9860703945159912 ,&Purity/3FRt3SRqqt4FBt/event_proportion0 particle/accuracy_3_of_71 7 Purity/*FRt2SRqqt3FBt/FBt_purity0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity0.6238994598388672 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity0.20127779245376587 6 Purity/1FRt2SRqqt1FBt/FRt_purity0.243778795003891 @&Purity/1FRt2SRqqt2FBt/event_proportion0.0010080644860863686 ("Purity/2FRt*SRqqt3FBt/event_purity0 7 Purity/*FRt*SRqqt3FBt/FBt_purity0.8651341795921326 8 Purity/3FRt1SRqqt1FBt/FRt_purity0.23076923191547394 ,&Purity/4FRt3SRqqt2FBt/event_proportion0 ,&Purity/4FRt4SRqqt*FBt/event_proportion0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity1 ("Purity/4FRt0SRqqt*FBt/event_purity0 :"Purity/1FRt*SRqqt2FBt/event_purity0.06752701103687286 ?&Purity/1FRt0SRqqt1FBt/event_proportion0.049773186445236206 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity0.2422444075345993 7 Purity/2FRt0SRqqt1FBt/FRt_purity0.3164682388305664 ,&Purity/4FRt4SRqqt3FBt/event_proportion0 :"Purity/0FRt*SRqqt0FBt/event_purity0.14738963544368744 >&Purity/0FRt2SRqqt*FBt/event_proportion0.05361643061041832 7 Purity/0FRt2SRqqt*FBt/FBt_purity0.9685100317001343 , Purity/1FRt*SRqqt3FBt/FBt_purity0.90625 ?&Purity/1FRt3SRqqt*FBt/event_proportion0.009135584346950054 ("Purity/3FRt0SRqqt1FBt/event_purity0 @&Purity/4FRt1SRqqt0FBt/event_proportion6.300403038039804e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity0.3750000298023224 ?&Purity/0FRt2SRqqt0FBt/event_proportion0.043283771723508835 +jet/accuracy_3_of_40.03944425284862518 7 Purity/*FRt4SRqqt0FBt/FRt_purity0.3750000298023224 ("Purity/0FRt3SRqqt*FBt/event_purity0 & Purity/3FRt2SRqqt*FBt/FBt_purity1 ("Purity/1FRt3SRqqt2FBt/event_purity0 A&Purity/1FRt*SRqqt4FBt/event_proportion0.00012600806076079607 ?&Purity/2FRt*SRqqt1FBt/event_proportion0.021169355139136314 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity0.29449525475502014 particle/accuracy_1_of_71 8"Purity/3FRt3SRqqt*FBt/SRqqt_purity0.261904776096344 7 Purity/*FRt0SRqqt3FBt/FBt_purity0.8771826028823853 @&Purity/*FRt2SRqqt3FBt/event_proportion6.300403038039804e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion0.0013230846961960196 ,&Purity/0FRt4SRqqt4FBt/event_proportion0 & Purity/3FRt*SRqqt1FBt/FBt_purity1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity0.6666666865348816 6 Purity/*FRt1SRqqt*FBt/FBt_purity0.984598696231842 ,&Purity/1FRt2SRqqt4FBt/event_proportion0 :"Purity/2FRt*SRqqt*FBt/event_purity0.00453285314142704 ?&Purity/2FRt3SRqqt*FBt/event_proportion0.004032257944345474 ,&Purity/2FRt4SRqqt1FBt/event_proportion0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity0.15763889253139496 particle/accuracy_2_of_71 & Purity/2FRt*SRqqt3FBt/FBt_purity1 8 Purity/*FRt*SRqqt1FBt/FRt_purity0.33684012293815613 6 Purity/*FRt2SRqqt*FBt/FBt_purity0.984845757484436 ;"Purity/*FRt2SRqqt1FBt/event_purity0.001792114693671465 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity0.18784432113170624 @&Purity/0FRt4SRqqt*FBt/event_proportion0.0005040322430431843 @&Purity/1FRt4SRqqt0FBt/event_proportion0.0003780241822823882 @&Purity/3FRt0SRqqt*FBt/event_proportion0.0027091733645647764 ("Purity/1FRt4SRqqt*FBt/event_purity0 =&Purity/1FRt*SRqqt*FBt/event_proportion0.4366179406642914 ,&Purity/1FRt4SRqqt1FBt/event_proportion0 8"Purity/0FRt1SRqqt3FBt/SRqqt_purity0.699999988079071 :"Purity/0FRt1SRqqt1FBt/event_purity0.12495119869709015 8 Purity/1FRt*SRqqt1FBt/FRt_purity0.33061540126800537 ("Purity/2FRt2SRqqt*FBt/event_purity0 +jet/accuracy_2_of_40.26650553941726685 & Purity/*FRt0SRqqt4FBt/FRt_purity0 ( Purity/2FRt2SRqqt2FBt/FRt_purity0.5 7 Purity/*FRt3SRqqt1FBt/FRt_purity0.2944444417953491 >&Purity/0FRt0SRqqt1FBt/event_proportion0.08606351166963577 6 Purity/1FRt1SRqqt*FBt/FRt_purity0.272990882396698 8 Purity/1FRt1SRqqt0FBt/FRt_purity0.25753429532051086 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity0 & Purity/2FRt0SRqqt3FBt/FBt_purity1 >&Purity/*FRt0SRqqt1FBt/event_proportng3͔Dion0.14591734111309052 ("Purity/3FRt1SRqqt1FBt/event_purity0 ,&Purity/4FRt2SRqqt3FBt/event_proportion0 >&Purity/*FRt2SRqqt1FBt/event_proportion0.02255544438958168 :"Purity/0FRt*SRqqt2FBt/event_purity0.25910335779190063 7 Purity/1FRt0SRqqt*FBt/FBt_purity0.9760212302207947 @&Purity/1FRt3SRqqt1FBt/event_proportion0.0011340725468471646 & Purity/*FRt*SRqqt4FBt/FRt_purity0 7 Purity/3FRt0SRqqt2FBt/FRt_purity0.3333333432674408 >&Purity/2FRt*SRqqt*FBt/event_proportion0.10244455933570862 7 Purity/2FRt*SRqqt1FBt/FRt_purity0.3648886978626251 ,&Purity/4FRt3SRqqt3FBt/event_proportion0 ,&Purity/4FRt4SRqqt4FBt/event_proportion0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity0 7 Purity/*FRt0SRqqt1FBt/FRt_purity0.3519068658351898 =&Purity/*FRt2SRqqt*FBt/event_proportion0.1332535296678543 ,&Purity/0FRt1SRqqt4FBt/event_proportion0 & Purity/2FRt1SRqqt1FBt/FBt_purity1 ,&Purity/3FRt*SRqqt3FBt/event_proportion0 & Purity/1FRt*SRqqt4FBt/FRt_purity0 9"Purity/*FRt0SRqqt0FBt/event_purity0.1530666947364807 7 Purity/*FRt1SRqqt2FBt/FBt_purity0.9673060774803162 ,&Purity/0FRt3SRqqt2FBt/event_proportion0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity0.2797619104385376 @&Purity/3FRt1SRqqt0FBt/event_proportion0.0025201612152159214 ,&Purity/4FRt3SRqqt1FBt/event_proportion0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity0 9"Purity/*FRt0SRqqt*FBt/event_purity0.3726472556591034 >&Purity/1FRt2SRqqt*FBt/event_proportion0.05727066472172737 ,&Purity/2FRt1SRqqt3FBt/event_proportion0 *loss/val_total_loss46.220626354312046 >&Purity/0FRt*SRqqt2FBt/event_proportion0.03944052383303642 ,&Purity/2FRt*SRqqt4FBt/event_proportion0 @&Purity/4FRt*SRqqt0FBt/event_proportion0.0001890120911411941 @&Purity/4FRt2SRqqt0FBt/event_proportion6.300403038039804e-05 jet/accuracy_6_of_70 :"Purity/*FRt*SRqqt1FBt/event_purity0.38478487730026245 7 Purity/0FRt*SRqqt1FBt/FBt_purity0.9939448833465576 :"Purity/1FRt3SRqqt1FBt/SRqqt_purity0.17777776718139648 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity0.1763174682855606 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity0.5333333611488342 ("Purity/3FRt1SRqqt0FBt/event_purity0 ,&Purity/0FRt4SRqqt3FBt/event_proportion0 ,&Purity/1FRt3SRqqt4FBt/event_proportion0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity0.2040543556213379 ,&Purity/4FRt*SRqqt4FBt/event_proportion0 7 Purity/0FRt2SRqqt2FBt/FBt_purity0.9807692170143127 ) Purity/1FRt0SRqqt4FBt/FBt_purity0.75 ("Purity/1FRt1SRqqt4FBt/event_purity0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion0.01827116869390011 ;"Purity/2FRt0SRqqt0FBt/event_purity0.003225806402042508 ("Purity/3FRt0SRqqt*FBt/event_purity0 ("Purity/3FRt2SRqqt*FBt/event_purity0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity0.24094201624393463 ,&Purity/4FRt*SRqqt3FBt/event_proportion0 ,&Purity/4FRt3SRqqt*FBt/event_proportion0 ("Purity/0FRt4SRqqt0FBt/event_purity0 @&Purity/*FRt4SRqqt0FBt/event_proportion0.0012600806076079607 jet/accuracy_4_of_70 ("Purity/3FRt*SRqqt2FBt/event_purity0 9"Purity/*FRt*SRqqt*FBt/event_purity0.1813197284936905 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity0.21792234480381012 @&Purity/1FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion0.0024571572430431843 & Purity/0FRt4SRqqt1FBt/FBt_purity1 particle/accuracy_1_of_61 particle/accuracy_5_of_71 ( Purity/2FRt0SRqqt3FBt/FRt_purity0.5 8 Purity/1FRt3SRqqt1FBt/FRt_purity0.35555556416511536 8 Purity/2FRt2SRqqt*FBt/FRt_purity0.25309279561042786 8 Purity/2FRt2SRqqt0FBt/FRt_purity0.22267256677150726 ,&Purity/4FRt3SRqqt4FBt/event_proportion0 jet/accuracy_5_of_70 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity1 7 Purity/1FRt1SRqqt2FBt/FRt_purity0.3263605237007141 ("Purity/3FRt0SRqqt0FBt/event_purity0 *jet/accuracy_2_of_30.1745252013206482 8"Purity/3FRt3SRqqt0FBt/SRqqt_purity0.261904776096344 :"Purity/*FRt*SRqqt0FBt/event_purity0.09324809163808823 :"Purity/*FRt*SRqqt3FBt/event_purity0.08448275923728943 8 Purity/*FRt0SRqqt3FBt/FRt_purity0.36666667461395264 ?&Purity/0FRt3SRqqt0FBt/event_proportion0.006111391354352236 6 Purity/1FRt0SRqqt2FBt/FBt_purity0.958686888217926 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity0 7"Purity/*FRt1SRqqt3FBt/SRqqt_purity0.69047611951828 6 Purity/0FRt1SRqqt3FBt/FBt_purity0.800000011920929 @&Purity/0FRt*SRqqt3FBt/event_proportion0.0030871976632624865 @&Purity/3FRt*SRqqt2FBt/event_proportion0.0003150201519019902 ,&Purity/4FRt*SRqqt1FBt/event_proportion0 ,&Purity/4FRt1SRqqt2FBt/event_proportion0 ( Purity/2FRt*SRqqt3FBt/FRt_purity0.5 :"Purity/*FRt*SRqqt2FBt/event_purity0.17645971477031708 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity0.1798771321773529 8 Purity/*FRt*SRqqt3FBt/FRt_purity0.38235294818878174 =&Purity/*FRt0SRqqt*FBt/event_proportion0.3558467626571655 >&Purity/*FRt2SRqqt0FBt/event_proportion0.10855595022439957 ("Purity/2FRt1SRqqt1FBt/event_purity0 ("Purity/4FRt1SRqqt0FBt/event_purity0 ("Purity/4FRt2SRqqt*FBt/event_purity0 ("Purity/0FRt0SRqqt0FBt/event_purity1 ,&Purity/2FRt3SRqqt2FBt/event_proportion0 @&Purity/3FRt0SRqqt0FBt/event_proportion0.0015120967291295528 ,&Purity/4FRt0SRqqt3FBt/event_proportion0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity0.15625 ) Purity/2FRt3SRqqt1FBt/FRt_purity0.25 jet/accuracy_3_of_70 particle/accuracy_4_of_71 ,&Purity/*FRt4SRqqt2FBt/event_proportion0 * Purity/1FRt*SRqqt3FBt/FRt_purity0.375 8 Purity/3FRt0SRqqt*FBt/FRt_purity0.21969695389270782 8 Purity/3FRt0SRqqt0FBt/FRt_purity0.11437907814979553 ("Purity/2FRt4SRqqt*FBt/event_purity0 7 Purity/*FRt2SRqqt2FBt/FBt_purity0.9824560880661011 ) Purity/*FRt1SRqqt4FBt/FBt_purity0.75 7 Purity/*FRt*SRqqt2FBt/FBt_purity0.9703595042228699 ("Purity/*FRt3SRqqt0FBt/event_purity0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity0.218288853764534 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity0.21220026910305023 ("Purity/0FRt2SRqqt2FBt/event_purity0 @&Purity/*FRt0SRqqt4FBt/event_proportion0.0003150201519019902 ,&Purity/2FRt2SRqqt4FBt/event_proportion0 ,&Purity/3FRt2SRqqt3FBt/event_proportion0 7validation_average_jet_accuracy0.33618563413619995 & Purity/2FRt3SRqqt*FBt/FBt_purity1 @&Purity/*FRt3SRqqt2FBt/event_proportion6.300403038039804e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity0.9612084031105042 A&Purity/1FRt1SRqqt3FBt/event_proportion0.00025201612152159214 ?&Purity/2FRt*SRqqt2FBt/event_proportion0.005418346729129553 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity0.22369182109832764 ?&Purity/2FRt2SRqqt*FBt/event_proportion0.019468246027827263 ,&Purity/2FRt2SRqqt3FBt/event_proportion0 ,&Purity/4FRt0SRqqt4FBt/event_proportion0 9"Purity/*FRt1SRqqt1FBt/event_purity0.0736161395907402 @&Purity/0FRt3SRqqt1FBt/event_proportion0.0006300403038039804 & Purity/2FRt2SRqqt1FBt/FBt_purity1 ( Purity/*FRt3SRqqt2FBt/FBt_purity0.5 ("Purity/4FRt1SRqqt*FBt/event_purity0 ?&Purity/*FRt0SRqqt3FBt/event_proportion0.003717237850651145 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity0.20665742456912994 >&Purity/0FRt0SRqqt*FBt/event_proportion0.11857359111309052 & Purity/3FRt0SRqqt1FBt/FBt_purity1 jet/accuracy_5_of_60 >&Purity/1FRt0SRqqt*FBt/event_proportion0.19745463132858276 ?&Purity/2FRt0SRqqt*FBt/event_proportion0.037046369165182114 particle/accuracy_2_of_21 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity0 >&Purity/1FRt0SRqqt0FBt/event_proportion0.13142640888690948 7 Purity/2FRt*SRqqt2FBt/FBt_purity0.9855555295944214 7 Purity/2FRt1SRqqt2FBt/FRt_purity0.3179824650287628 ("Purity/*FRt0SRqqt4FBt/event_purity0 jet/accuracy_7_of_70 7 Purity/*FRt0SRqqt2FBt/FBt_purity0.9714804887771606 7 Purity/1FRt0SRqqt3FBt/FRt_purity0.3571428656578064 , Purity/1FRt3SRqqt*FBt/FBt_purity0.96875 ;"Purity/2FRt*SRqqt0FBt/event_purity0.001240694778971374 @&Purity/2FRt3SRqqt1FBt/event_proportion0.0003780241822823882 ("Purity/3FRt0SRqqt2FBt/event_purity0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity0 ("Purity/2FRt2SRqqt2FBt/event_purity0 >&Purity/0FRt1SRqqt1FBt/event_proportion0.05185231938958168 ("Purity/2FRt3SRqqt0FBt/event_purity0 7 Purity/1FRt2SRqqt2FBt/FBt_purity0.9861111044883728 @&Purity/*FRt*SRqqt4FBt/event_proportion0.0003780241822823882 =&Purity/*FRt1SRqqt1FBt/event_proportion0.1001134067773819 6 Purity/*FRt0SRqqt4FBt/FBt_purity0.699999988079071 & Purity/3FRt1SRqqt2FBt/FBt_purity1 6 Purity/2FRt0SRqqt2FBt/FBt_purity0.976190447807312 particle/accuracy_5_of_61 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity0.39841270446777344 @&Purity/4FRt0SRqqt0FBt/event_proportion6.300403038039804e-05 7"Purity/0FRt0SRqqt*FBt/event_purity0.81014484167099 7 Purity/0FRt1SRqqt*FBt/FBt_purity0.9822250604629517 @&Purity/1FRt4SRqqt*FBt/event_proportion0.0003780241822823882 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity0.5350877046585083 ?&Purity/2FRt0SRqqt0FBt/event_proportion0.024571573361754417 ("Purity/2FRt1SRqqt2FBt/event_purity0 ) Purity/1FRt*SRqqt4FBt/FBt_purity0.75 ("Purity/1FRt2SRqqt*FBt/event_purity0 ,&Purity/3FRt1SRqqt4FBt/event_proportion0 @&Purity/3FRt2SRqqt1FBt/event_proportion0.0003780241822823882 +jet/accuracy_2_of_20.08100736886262894 ) Purity/*FRt1SRqqt3FBt/FRt_purity0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity0.3333333432674408 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity0.1797143965959549 ?&Purity/2FRt1SRqqt2FBt/event_proportion0.001890120911411941 7 Purity/*FRt*SRqqt4FBt/FBt_purity0.7083333134651184 ("Purity/*FRt4SRqqt*FBt/event_purity0 :"Purity/0FRt*SRqqt*FBt/event_purity0.32967686653137207 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity0.31308239698410034 8 Purity/3FRt*SRqqt2FBt/FRt_purity0.20000000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity0 & Purity/4FRt2SRqqt*FBt/FRt_purity0 & Purity/4FRt2SRqqt0FBt/FRt_purity0 ,&Purity/*FRt2SRqqt4FBt/event_proportion0 7 Purity/1FRt*SRqqt*FBt/FRt_purity0.2545413076877594 8 Purity/1FRt*SRqqt0FBt/FRt_purity0.21964110434055328 @&Purity/3FRt2SRqqt0FBt/event_proportion0.0024571572430431843 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity0.16021355986595154 & Purity/1FRt3SRqqt2FBt/FRt_purity1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity0.19120079278945923 8 Purity/*FRt3SRqqt0FBt/FRt_purity0.17853040993213654 7 Purity/1FRt1SRqqt1FBt/FRt_purity0.3108481764793396 7 Purity/2FRt1SRqqt*FBt/FBt_purity0.9964157938957214 ("Purity/3FRt3SRqqt*FBt/event_purity0 8 Purity/*FRt*SRqqt*FBt/FRt_purity0.25110387802124023 7 Purity/*FRt*SRqqt0FBt/FRt_purity0.2141963541507721 A&Purity/0FRt4SRqqt0FBt/event_proportion0.00044102821266278625 @&Purity/0FRt4SRqqt1FBt/event_proportion6.300403038039804e-05 ("Purity/1FRt3SRqqt0FBt/event_purity0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity0.5 7 Purity/1FRt0SRqqt1FBt/FBt_purity0.9929931163787842 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity0.2331511527299881 :"Purity/2FRt0SRqqt2FBt/event_purity0.01785714365541935 jet/accuracy_4_of_50 particle/accuracy_1_of_41 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity0.19924761354923248 >&Purity/*FRt0SRqqt0FBt/event_proportion0.15757308900356293 >&Purity/0FRt*SRqqt0FBt/event_proportion0.21931703388690948 :"Purity/0FRt0SRqqt3FBt/event_purity0.13333334028720856 ?&Purity/1FRt3SRqqt0FBt/event_proportion0.007938507944345474 7 Purity/2FRt*SRqqt*FBt/FRt_purity0.2479925900697708 8 Purity/2FRt*SRqqt0FBt/FRt_purity0.20610344409942627 + Purity/0FRt*SRqqt4FBt/FBt_purity0.6875 8 Purity/*FRt0SRqqt*FBt/FRt_purity0.24298721551895142 8 Purity/*FRt0SRqqt0FBt/FRt_purity0.17775459587574005 ?&Purity/1FRt2SRqqt1FBt/event_proportion0.008946572430431843 ,&Purity/1FRt2SRqqt3FBt/event_proportion0 ("Purity/2FRt2SRqqt1FBt/event_purity0 ,particle/sensitivity0.24017666280269623 & Purity/1FRt1SRqqt4FBt/FRt_purity0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity0.31315839290618896 7 Purity/*FRt2SRqqt1FBt/FBt_purity0.9886924624443054 ;"Purity/1FRt1SRqqt*FBt/event_purity0.011674611829221249 ("Purity/1FRt3SRqqt1FBt/event_purity0 ("Purity/3FRt*SRqqt1FBt/event_purity0 ?&Purity/1FRt*SRqqt2FBt/event_proportion0.022303426638245583 ,&Purity/3FRt*SRqqt4FBt/event_proportion0 ("Purity/3FRt3SRqqt0FBt/event_purity0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity0.3857017159461975 8 Purity/2FRt0SRqqt*FBt/FRt_purity0.22881494462490082 7 Purity/2FRt0SRqqt0FBt/FRt_purity0.1658840775489807 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity0.31739628314971924 & Purity/3FRt*SRqqt*FBt/FBt_purity1 7 Purity/1FRt1SRqqt3FBt/FBt_purity0.8333333730697632 7 Purity/*FRt1SRqqt1FBt/FBt_purity0.9938223361968994 >&Purity/2FRt*SRqqt0FBt/event_proportion0.07579384744167328 ,&Purity/2FRt1SRqqt4FBt/event_proportion0 # _timestamp1.78846936008347e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion6.300403038039804e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity0.9736158847808838 7 Purity/0FRt0SRqqt3FBt/FBt_purity0.8822223544120789 ,&Purity/0FRt2SRqqt4FBt/event_proportion0 8 Purity/1FRt2SRqqt*FBt/FRt_purity0.22948196530342102 8 Purity/1FRt2SRqqt0FBt/FRt_purity0.22213882207870483 @&Purity/2FRt*SRqqt3FBt/event_proportion6.300403038039804e-05 ("Purity/2FRt2SRqqt0FBt/event_purity0 >&Purity/0FRt*SRqqt*FBt/event_proportion0.40996724367141724 ,&Purity/2FRt4SRqqt4FBt/event_proportion0 7 Purity/3FRt1SRqqt*FBt/FRt_purity0.2222222536802292 8 Purity/3FRt1SRqqt0FBt/FRt_purity0.20694446563720703 ("Purity/3FRt2SRqqt0FBt/event_purity0 ,&Purity/4FRt2SRqqt4FBt/event_proportion0 particle/accuracy_2_of_41 & Purity/3FRt2SRqqt1FBt/FBt_purity1 jet/accuracy_5_of_50 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity0.1785714328289032 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity0.2083333283662796 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity0.9792627096176147 -"Purity/3FRt1SRqqt0FBt/SRqqt_purity0.3125 particle/accuracy_4_of_51 jet/accuracy_2_of_70 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity0.21966226398944855 particle/accuracy_3_of_31 particle/accuracy_4_of_61 ) Purity/4FRt1SRqqt*FBt/FRt_purity0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity0.25 ("Purity/4FRt2SRqqt0FBt/event_purity0 :"Purity/0FRt*SRqqt3FBt/event_purity0.12345679849386215 9"Purity/1FRt0SRqqt3FBt/event_purity0.0714285746216774 ,&Purity/3FRt0SRqqt4FBt/event_proportion0 @&Purity/4FRt0SRqqt*FBt/event_proportion6.300403038039804e-05 ("Purity/*FRt4SRqqt1FBt/event_purity0 & Purity/3FRt1SRqqt1FBt/FBt_purity1 ,&Purity/3FRt2SRqqt2FBt/event_proportion0 ,&Purity/4FRt*SRqqt2FBt/event_proportion0 +particle/specificity0.9482179284095764 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity1 :"Purity/*FRt0SRqqt3FBt/event_purity0.08749999850988388 :"Purity/0FRt0SRqqt2FBt/event_purity0.32632818818092346 :"Purity/1FRt0SRqqt2FBt/event_purity0.10900913923978806 ?&Purity/1FRt1SRqqt2FBt/event_proportion0.006111391354352236 7 Purity/1FRt2SRqqt1FBt/FBt_purity0.9873272180557251 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity0.28703704476356506 8 Purity/3FRt2SRqqt*FBt/FRt_purity0.20370370149612427 8 Purity/3FRt2SRqqt0FBt/FRt_purity0.20410628616809845 7 Purity/*FRt1SRqqt*FBt/FRt_purity0.2667010724544525 8 Purity/*FRt1SRqqt0FBt/FRt_purity0.24521878361701965 9"Purity/1FRt0SRqqt0FBt/event_purity0.1830933839082718 particle/accuracy_6_of_61 * Purity/0FRt3SRqqt*FBt/FBt_purity0.875 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity0 & Purity/2FRt0SRqqt1FBt/FBt_purity1 +jet/accuracy_4_of_60.02631578966975212 particle/accuracy_6_of_71 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity0.5 ,&Purity/1FRt4SRqqt4FBt/event_proportion0 particle/accuracy_4_of_41 :"Purity/0FRt2SRqqt2FBt/SRqqt_purity0.32692307233810425 _runtime4250.126188169 ?&Purity/2FRt0SRqqt2FBt/event_proportion0.003465221729129553 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity0.23389437794685364 *jet/accuracy_1_of_60.9473684430122375 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity0.34633129835128784 @&Purity/0FRt2SRqqt2FBt/event_proportion0.0010080644860863686 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity0.1766233742237091 ?&Purity/3FRt*SRqqt*FBt/event_proportion0.009954636916518211 & Purity/*FRt4SRqqt*FBt/FBt_purity1 :"Purity/0FRt1SRqqt*FBt/event_purity0.16795898973941803 8 Purity/3FRt*SRqqt1FBt/FRt_purity0.25757575035095215 ,&Purity/0FRt3SRqqt3FBt/event_proportion0 8 Purity/1FRt0SRqqt*FBt/FRt_purity0.24723216891288757 7 Purity/1FRt0SRqqt0FBt/FRt_purity0.1830933839082718 =&Purity/1FRt1SRqqt*FBt/event_proportion0.1723790317773819 :"Purity/2FRt*SRqqt2FBt/event_purity0.01666666753590107 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity0.5175438523292542 7 Purity/*FRt*SRqqt1FBt/FBt_purity0.9941746592521667 A&Purity/0FRt*SRqqt4FBt/event_proportion0.00025201612152159214 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity0.21259558200836182 ,&Purity/3FRt3SRqqt3FBt/event_proportion0 ,&Purity/3FRt4SRqqt1FBt/event_proportion0 ("Purity/*FRt3SRqqt2FBt/event_purity0 <"Purity/*FRt2SRqqt0FBt/event_purity0.0050620995461940765 ?&Purity/*FRt3SRqqt1FBt/event_proportion0.002142137149348855 6 Purity/*FRt3SRqqt1FBt/FBt_purity0.976190447807312h?y(:  gpu.0.gpu63.0 gpu.0.memory15+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts325.437' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.0874 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0 x(: ϴ disk.sda.out 131.23828125 network.sent 5.609674e+06) disk./.usagePercent77.21530948402643 disk.nvme0n1.in 9602.921875& disk.nvme0n1.out97412.41748046875 disk.sda3.in0 disk.sda3.out 131.23828125 disk.sda2.in0 network.recv 1.376691e+06 disk.sda1.in0 disk.sda1.out0 disk.sda2.out0! disk.nvme0c0n1.in 9602.921875( disk.nvme0c0n1.out97412.41748046875 disk.drbd1260.in0$ memory_percent88.91596794128418 disk.sda.in0% disk./.usageGB1150.5639114379883 disk.dm-0.in 9602.921875! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266 cpu25.34311613421027 proc.cpu.threads17# disk.dm-0.out97412.41748046875 disk.dm-1.in0 disk.dm-1.out 131.23828125 disk.drbd1260.out0.59375( proc.memory.availableMB 1816.0078125y(:  disk.sda3.in0 disk.sda2.out0 disk.drbd1260.in0 disk.drbd1260.out 0.6015625 disk.dm-0.in 9738.73046875# disk.dm-0.out97509.41943359375( proc.memory.availableMB 1816.6015625 cpu25.34322966289769 network.recv 1.382716e+06) disk./.usagePercent77.20852118824853 disk.dm-1.in0 disk.dm-1.out 131.703125 proc.cpu.threads17$ disk./.usageGB1150.462760925293 disk.sda.in0( disk.nvme0c0n1.out97509.41943359375! disk.nvme0n1.in 9738.73046875 disk.sda3.out 131.703125$ memory_percent88.91234397888184! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266 network.sent 5.714586e+06 disk.sda.out 131.703125 disk.sda2.in0# disk.nvme0c0n1.in 9738.73046875& disk.nvme0n1.out97509.41943359375 disk.sda1.in0 disk.sda1.out0ݑ(: ϓ gpu.0.gpu10.0 gpu.0.memory5+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp40.0 gpu.0.powerWatts117.005' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent23.401 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0|(:  disk.sda2.out0 cpu25.34359405803198 network.sent 5.718425e+06 network.recv 1.38683e+06 disk.dm-1.in0! disk.nvme0n1.in 9738.73046875& disk.nvme0n1.out97773.03271484375 disk.sda1.in0) proc.memory.availableMB 1816.34765625 proc.cpu.threads17( disk.nvme0c0n1.out97773.03271484375 disk.dm-0.in 9738.73046875# disk.dm-0.out97773.03271484375 disk.sda1.out0 disk.sda3.in0! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266( disk./.usagePercent77.2085232363088$ disk./.usageGB1150.462791442871 disk.drbd1260.out 0.6015625 disk.sda3.out 132.18359375 disk.sda.in0 disk.sda.out 132.18359375 disk.sda2.in0" memory_percent88.913893699646# disk.nvme0c0n1.in 9738.73046875 disk.dm-1.out 132.18359375 disk.drbd1260.in0 q5(: Ŏ gpu.0.gpu97.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp45.0 gpu.0.powerWatts242.846' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.5692 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts234.864' gpu.0.enforcedPowerLimitWatts500.0( gpu.0.powerPercent46.97280000000001 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0=:K(:  disk.sda1.in0 disk.sda.in0 disk.sda.out 132.69921875 disk.sda2.in0 disk.sda2.out0" memory_percent88.913893699646 proc.cpu.threads17 network.recv 1.390387e+06) disk./.usagePercent77.20825596444305% disk./.usageGB1150.4588088989258 disk.dm-0.in 9738.73046875 disk.sda1.out0 disk.sda3.out 132.69921875 disk.dm-1.out 132.69921875 disk.drbd1260.in0 network.sent 5.723447e+06 disk.sda3.in0# disk.nvme0c0n1.in 9738.73046875 disk.dm-1.in0) proc.memory.availableMB 1816.34765625! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266 cpu25.343908562825355! disk.nvme0n1.in 9738.73046875# disk.dm-0.out98145.84326171875( disk.nvme0c0n1.out98145.84326171875 disk.drbd1260.out 0.65234375& disk.nvme0n1.out98145.84326171875J 2- _wandbruntime4305  _runtime4305CF & lr-AdamW/pg20.0008735362282945423 trainer/global_step2449 % _timestamp1.7884694157177129e+09 _runtime4305.758342237 _step196 & lr-AdamW/pg10.0008735362282945423; trainer/global_step2449 % _timestamp1.7884694157177129e+09 _runtime4305.758342237 _step196 & lr-AdamW/pg10.0008735362282945423 & lr-AdamW/pg20.0008735362282945423֔X 2- _wandbruntime4305  _runtime43057T$O ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated534825650808832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated35332190 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5692651326464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated540518302135296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed540515760344064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed540515760344064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated540518302135296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed28382081 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed108673984992768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed46097545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5682658814405 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed35332184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated28382532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed115937468542464 % _timestamp1.7884694157191951e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated35332190 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated46117229 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 _runtime4305.762597761 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5690134963200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5692651326464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed513040212816272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5685173182817 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated518725411427057 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed46097545 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated17509051 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current451 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7263483549696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed10873478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated115937554482688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed534825625380864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed518722871630677 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 trainer/global_step2449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5690134963200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed17508603 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 451453440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed35332184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_(bytes.large_pool.current29097984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated81449419 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated513040238244240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated108674014090752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 _step197 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated46117229G'O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5690134963200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated 5692651326464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed513040212816272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5685173182817 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated518725411427057 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed46097545 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated17509051 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current451 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7263483549696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed10873478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated115937554482688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed534825625380864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed518722871630677 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 trainer/global_step2449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5690134963200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed17508603 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 451453440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed35332184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated81449419 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated513040238244240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated108674014090752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 _step197 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated46117229 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated534825650808832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated35332190 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5692651326464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated540518302135296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed540515760344064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed540515760344064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated540518302135296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed28382081 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed108673984992768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed46097545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5682658814405 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed35332184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated28382532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed115937468542464 % _timestamp1.7884694157191951e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated35332190 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated46117229 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 _runtime4305.762597761 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0b2- _wandbruntime4307  _runtime4307KO'M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed35346065 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated28393782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed46115896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10877813 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated513242766953824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3204448256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated108718075460096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated35346071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed535036615756288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5694926556160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1203 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56828928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed46115896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step2449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed540729025935872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed28393323 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated46135606 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated540731567740416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated518930212391703 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5684931069259 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed535036615756288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated35346071 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 # _timestamp1.78846941719753e+09 _runtime4307.238398356 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated81481677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0^''N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated46135606 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated540731567740416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated518930212391703 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed 5684931069259 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated2146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed535036615756288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated35346071 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak 34275006464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 # _timestamp1.78846941719753e+09 _runtime4307.238398356 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated81481677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed35346065 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated28393782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed46115896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current1528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated10877813 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated513242766953824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak 3204448256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current 40265318400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current 3204448256 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated108718075460096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated35346071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed535036615756288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5694926556160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak 960105472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1203 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56828928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed46115896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 trainer/global_step2449 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed540729025935872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current25427968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed28393323 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 25418045442- _wandbruntime4307  _runtime4307{\ /loss/FRt2/assignment_loss3.993732329727947 2loss/SRqqt1/assignment_loss1.1388888492418283 2loss/SRqqt2/assignment_loss2.0077278022486045 1loss/FBt4/assignment_loss0.21991859891244936 /loss/FRt2/detection_loss2.5825692769409954 /loss/FRt4/detection_loss2.1765158775037734 0loss/SRqqt2/detection_loss2.153793806412134 .loss/FBt2/detection_loss2.924205740292867 0loss/FRt1/assignment_loss1.5898278086650661 2loss/SRqqt4/assignment_loss1.7140613157749176 1loss/FBt2/assignment_loss0.13872297055786476 .loss/FBt3/detection_loss3.183688461780548 epoch4 % _timestamp1.7884694171999495e+09 _step199 1loss/SRqqt3/assignment_loss2.116748065207944 1loss/FBt1/assignment_loss0.15290539752228885 trainer/global_step2449 /loss/FRt4/assignment_loss3.070813025893836 1loss/SRqqt3/detection_loss2.5853178704326805 1loss/SRqqt4/detection_loss1.6566103484630585 /loss/FBt4/detection_loss2.6977663536866507 /loss/FRt1/detection_loss1.9303572705787857 .loss/FRt3/detection_loss3.018463695049286 _runtime4307.240244587 /loss/FRt3/assignment_loss4.006168417497115 1loss/FBt3/assignment_loss0.11387071309062387 1loss/SRqqt1/detection_loss0.9812143807948539 /loss/FBt1/detection_loss0.7543230982430812 &loss/total_loss46.908211474519206?$8 /loss/FRt4/assignment_loss3.070813025893836 1loss/SRqqt3/detection_loss2.5853178704326805 1loss/SRqqt4/detection_loss1.6566103484630585 /loss/FBt4/detection_loss2.6977663536866507 /loss/FRt1. /detection_loss1.9303572705787857 .loss/FRt3/detection_loss3.018463695049286 _runtime4307.240244587 /loss/FRt3/assignment_loss4.006168417497115 1loss/FBt3/assignment_loss0.11387071309062387 1loss/SRqqt1/detection_loss0.9812143807948539 /loss/FBt1/detection_loss0.7543230982430812 &loss/total_loss46.908211474519206 /loss/FRt2/assignment_loss3.993732329727947 2loss/SRqqt1/assignment_loss1.1388888492418283 2loss/SRqqt2/assignment_loss2.0077278022486045 1loss/FBt4/assignment_loss0.21991859891244936 /loss/FRt2/detection_loss2.5825692769409954 /loss/FRt4/detection_loss2.1765158775037734 0loss/SRqqt2/detection_loss2.153793806412134 .loss/FBt2/detection_loss2.924205740292867 0loss/FRt1/assignment_loss1.5898278086650661 2loss/SRqqt4/assignment_loss1.7140613157749176 1loss/FBt2/assignment_loss0.13872297055786476 .loss/FBt3/detection_loss3.183688461780548 epoch4 % _timestamp1.7884694171999495e+09 _step199 1loss/SRqqt3/assignment_loss2.116748065207944 1loss/FBt1/assignment_loss0.15290539752228885 trainer/global_step2449fa((: ڟ gpu.0.gpu60.0 gpu.0.memory16+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp48.0 gpu.0.powerWatts331.578' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent66.3156 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0f+%q(:  network.recv 1.394989e+06# disk.nvme0c0n1.in 9738.73046875 disk.dm-0.in 9738.73046875 disk.sda3.out 133.12890625 disk.sda.out 133.12890625) proc.memory.availableMB 1815.77734375 proc.cpu.threads17 network.sent 5.729476e+06% disk./.usageGB1150.4333000183105( disk.nvme0c0n1.out98421.60498046875 disk.dm-1.out 133.12890625 disk.drbd1260.out 0.65234375 disk.sda1.out0 disk.sda2.in0$ memory_percent88.91737461090088& disk.nvme0n1.out98421.60498046875# disk.dm-0.out98421.60498046875 disk.sda1.in0 disk.sda3.in0 disk.sda.in0! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266! disk.nvme0n1.in 9738.73046875) disk./.usagePercent77.20654404206158 disk.dm-1.in0 disk.drbd1260.in0 disk.sda2.out0 cpu25.344286249116088Äi(: % disk./.usageGB1150.4625396728516! disk.nvme0n1.out 98714.421875 disk.dm-0.in 9899.359375! disk.nvme0c0n1.in 9899.359375 disk.dm-1.in0 disk.dm-1.out 133.60546875 network.sent 5.818423e+06) disk./.usagePercent77.20850633981155# disk.nvme0c0n1.out 98714.421875 disk.drbd1260.out0.65625! proc.memory.rssMB 3120.640625 cpu25.34441392404164 network.recv 1.400645e+06 disk.dm-0.out 98714.421875 disk.sda1.in0 disk.sda1.out0 disk.sda3.out 133.60546875 disk.sda2.in0 disk.drbd1260.in0$ memory_percent88.91282081604004 disk.nvme0n1.in 9899.359375 disk.sda3.in0 disk.sda2.out0( proc.memory.availableMB 1816.5234375* proc.memory.percent19.046878814697266 proc.cpu.threads17 disk.sda.in0 disk.sda.out 133.60546875~G(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts248.989' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent49.7978 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0{j(:  disk.sda1.in0 disk.sda1.out0* proc.memory.percent19.046878814697266 disk.dm-1.out 134.0546875) disk./.usagePercent77.20850377973619 disk.sda3.out 134.0546875 disk.sda.in0$ disk.nvme0c0n1.out 98809.7578125 disk.drbd1260.out0.65625 disk.nvme0n1.in 9899.359375 disk.dm-0.out 98809.7578125$ disk./.usageGB1150.462501525879 disk.sda2.out0 disk.drbd1260.in0" disk.nvme0n1.out 98809.7578125$ memory_percent88.91432285308838) proc.memory.availableMB 1816.27734375! proc.memory.rssMB 3120.640625 cpu25.344594030496953 disk.sda3.in0 disk.sda.out 134.0546875! disk.nvme0c0n1.in 9899.359375 disk.dm-1.in0 proc.cpu.threads17 network.sent 5.824054e+06 network.recv 1.404849e+06 disk.sda2.in0 disk.dm-0.in 9899.359375ϴ(:  gpu.0.gpu96.0 gpu.0.memory23+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp49.0 gpu.0.powerWatts338.266' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent67.6532 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0&t(: ֧ network.sent 5.827317e+06# disk.nvme0n1.out99084.23828125! disk.nvme0c0n1.in 9899.359375% disk.nvme0c0n1.out99084.23828125 disk.drbd1260.in0 disk.drbd1260.out 0.66015625$ memory_percent88.91284465789795% disk./.usageGB1150.5642585754395 disk.dm-0.in 9899.359375 disk.dm-0.out99084.23828125 disk.sda3.out 134.71484375 disk.sda2.out0 disk.dm-1.in0* proc.memory.percent19.046878814697266 cpu25.344721502218782 network.recv 1.408338e+06 disk.sda1.out0 disk.sda.in0 disk.sda.out 134.71484375 disk.dm-1.out 134.71484375! proc.memory.rssMB 3120.640625 proc.cpu.threads17) disk./.usagePercent77.21533278071206 disk.nvme0n1.in 9899.359375 disk.sda1.in0 disk.sda3.in0 disk.sda2.in0) proc.memory.availableMB 1816.51953125(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts240.726' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.1452 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0#.(: ΍ gpu.0.gpu98.0 gpu.0.memory24+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp48.0 gpu.0.powerWatts243.176' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent48.6352 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0NWq(:  disk.sda.out 135.06640625! proc.memory.rssMB 3120.640625 disk.drbd1260.out 0.66015625" disk.nvme0n1.out 99657.2421875 disk.sda2.in0 disk.dm-0.in 9899.359375 disk.sda1.in0 disk.sda3.in0$ memory_percent88.91584873199463 cpu25.344967206883215 network.recv 1.412608e+06) proc.memory.availableMB 1816.02734375* proc.memory.percent19.046878814697266 proc.cpu.threads17 network.sent 5.833286e+06) disk./.usagePercent77.22527534531935% disk./.usageGB1150.7124099731445 disk.sda2.out0! disk.nvme0c0n1.in 9899.359375 disk.dm-1.out 135.06640625 disk.drbd1260.in0 disk.sda1.out0$ disk.nvme0c0n1.out 99657.2421875 disk.dm-1.in0 disk.nvme0n1.in 9899.359375 disk.dm-0.out 99657.2421875 disk.sda3.out 135.06640625 disk.sda.in0q5!2- _wandbruntime4384  _runtime4384[$ _runtime4384.436762412 _step200 & lr-AdamW/pg10.0008585233516356583 & lr-AdamW/pg20.0008585233516356583 trainer/global_step2499 % _timestamp1.7884694943961525e+09d & lr-AdamW/pg10.0008585233516356583 & lr-AdamW/pg20.0008585233516356583 trainer/global_step2499 % _timestamp1.7884694943961525e+09 _runtime4384.436762412 _step2005!-2- _wandbruntime4384  _runtime4384h'O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed83044681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed551179067762688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed551179067762688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak882 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed36026234 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated523166673723440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5798813097525 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7409465206272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed11090078 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed528962947024585 trainer/global_step2499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed17854851 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 _runtime4384.438584688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed545375144152064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated47038131 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated36026240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 % _timestamp1.7884694943977377e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7409522048512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5796298729113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated528965486820965 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated551181609553920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated47038131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed523166648295472 _step201 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5803923610624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated551181609553920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed545375144152064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current451 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed28944929 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed118286518666240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5806439973888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated110877082557952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed36026234 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528Yb'O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed36026234 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed83044681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current56842240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed551179067762688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed551179067762688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak882 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed36026234 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated523166673723440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 5798813097525 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7409465206272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed11090078 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed528962947024585 trainer/global_step2499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed17854851 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 _runtime4384.438584688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed545375144152064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current85940224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated47038131 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated36026240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 % _timestamp1.7884694943977377e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current19684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7409522048512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5796298729113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated528965486820965 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated551181609553920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated47038131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed523166648295472 _step201 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 5803923610624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated2146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated551181609553920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed545375144152064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current451 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed28944929 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed118286518666240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5806439973888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated110877082557952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380%2- _wandbruntime4385  _runtime4385* ;M RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak23682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated11094413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak2146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated529170287785611 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated545586159955456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed36040115 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current458 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56828928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated1528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed17861670 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak 1400896000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed52916774798902WH3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated83096629 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5806198827008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed28956080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated523369202433024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated83096629 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85926912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated36040121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak882 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed523369177005056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated551394875159040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed545586134527488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated47056508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated118333585598464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 _step202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed83076913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5808715203584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed118333499671552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 7412441671168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5806198827008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed28956080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated523369202433024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated 3204448256 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated83096629 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current85926912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated36040121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak882 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak23981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current29097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed523369177005056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak 38561404416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated551394875159040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed545586134527488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated47056508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated118333585598464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 _step202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed83076913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated 5808715203584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed118333499671552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated 7412441671168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current 37060870144 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current458 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current 2541804544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current56828928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0[2- _wandbruntime4385  _runtime4385ccV epoch4 /loss/FRt2/assignment_loss3.606789200163599 trainer/global_step2499 1loss/FBt1/assignment_loss0.10787648150312293 1loss/SRqqt2/detection_loss2.6474667662068416 0loss/SRqqt4/detection_loss1.701591211651999 0loss/FRt3/assignment_loss3.6684037508635687 1loss/FBt4/assignment_loss0.14158435562014346 0loss/SRqqt1/detection_loss1.006293615172232 _step203 /loss/FBt1/detection_loss0.6087286938807066 /loss/FBt4/detection_loss2.2726178693771364 $ _timestamp1.788469495888441e+09 2loss/SRqqt1/assignment_loss1.2237373099179196 /loss/SRqqt2/assignment_loss2.0021243243756 2loss/SRqqt3/assignment_loss2.0020507870117825 1loss/SRqqt4/assignment_loss1.878768971101159 0loss/FBt3/assignment_loss0.6414503804116976 .loss/FRt4/detection_loss2.237926297915091 /loss/FBt3/detection_loss3.3991854190826416 %loss/total_loss46.88033074048397 /loss/FRt1/assignment_loss1.812003162599379 1loss/FBt2/assignment_loss0.22556062327283952 /loss/FRt2/detection_loss2.6809367040577494 1loss/SRqqt3/detection_loss2.3631847178936005 .loss/FBt2/detection_loss2.760989725589752 _runtime4385.929118436 /loss/FRt4/assignment_loss3.190703103301722 /loss/FRt1/detection_loss1.8509916877874764 /loss/FRt3/detection_loss2.8493655817262056OG[ /loss/FRt4/assignment_loss3.190703103301722 /loss/FRt1/detection_loss1.8509916877874764 /loss/FRt3/detection_loss2.8493655817262056 epoch4 /loss/FRt2/assignment_loss3.606789200163599 trainer/global_step2499 1loss/FBt1/assignment_loss0.10787648150312293 1loss/SRqqt2/detection_loss2.6474667662068416 0loss/SRqqt4/detection_loss1.701591211651999 0loss/FRt3/assignment_loss3.6684037508635687 1loss/FBt4/assignment_loss0.14158435562014346 0loss/SRqqt1/detection_loss1.006293615172232 _step203 /loss/FBt1/detection_loss0.6087286938807066 /loss/FBt4/detection_loss2.2726178693771364 $ _timestamp1.788469495888441e+09 2loss/SRqqt1/assignment_loss1.2237373099179196 /loss/SRqqt2/assignment_loss2.0021243243756 2loss/SRqqt3/assignment_loss2.0020507870117825 1loss/SRqqt4/assignment_loss1.878768971101159 0loss/FBt3/assignment_loss0.6414503804116976 .loss/FRt4/detection_loss2.237926297915091 /loss/FBt3/detection_loss3.3991854190826416 %loss/total_loss46.88033074048397 /loss/FRt1/assignment_loss1.812003162599379 1loss/FBt2/assignment_loss0.22556062327283952 /loss/FRt2/detection_loss2.6809367040577494 1loss/SRqqt3/detection_loss2.3631847178936005 .loss/FBt2/detection_loss2.760989725589752 _runtime4385.929118436H(: % disk./.usageGB1150.9392433166504 disk.sda.in0 disk.sda1.out0 disk.sda3.out 135.55859375 cpu25.345064594240004 proc.cpu.threads17) disk./.usagePercent77.24049832132788" disk.nvme0c0n1.in 10055.546875 disk.dm-1.out 135.55859375 disk.drbd1260.in0 disk.nvme0n1.in 10055.546875 disk.dm-0.in 10055.546875$ disk.dm-0.out100003.70556640625! proc.memory.rssMB 3120.640625 disk.sda2.out0) disk.nvme0c0n1.out100003.70556640625 disk.dm-1.in0 disk.drbd1260.out 0.6640625) proc.memory.availableMB 1816.23828125 network.recv 1.41583e+06 disk.sda.out 135.55859375 disk.sda2.in0' disk.nvme0n1.out100003.70556640625 disk.sda1.in0 disk.sda3.in0$ memory_percent88.91456127166748* proc.memory.percent19.046878814697266 network.sent 5.838621e+066ӓ(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts227.383' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent45.4766 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0n}(:  disk.sda3.out 136.1640625) disk.nvme0c0n1.out100121.02587890625 disk.dm-1.in0 disk.drbd1260.in0$ disk.dm-0.out100121.02587890625! proc.memory.rssMB 3120.640625 cpu25.34525962970209 network.sent 5.92504e+06 disk.sda.in0" disk.nvme0c0n1.in 10055.546875 disk.dm-0.in 10055.546875) proc.memory.availableMB 1815.98046875* proc.memory.percent19.046878814697266 proc.cpu.threads17% disk./.usageGB1150.9617233276367 disk.sda.out 136.1640625 disk.sda2.out0 disk.dm-1.out 136.1640625' disk.nvme0n1.out100121.02587890625 disk.sda1.in0 disk.sda1.out0$ memory_percent88.91613483428955 network.recv 1.421534e+06) disk./.usagePercent77.24200697372716 disk.sda2.in0 disk.drbd1260.out 0.6640625 disk.nvme0n1.in 10055.546875 disk.sda3.in0Rb(:  gpu.0.gpu38.0 gpu.0.memory9+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts325.239' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent65.0478 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0+Ps(: Ҁ$ memory_percent88.91465663909912! proc.memory.rssMB 3120.640625 network.sent 5.930077e+06 network.recv 1.42502e+06 disk.drbd1260.in0 disk.nvme0n1.in 10055.546875 disk.sda1.out0 disk.sda2.in0) disk.nvme0c0n1.out100299.25732421875 disk.dm-1.out136.6875) disk./.usagePercent77.24200697372716 disk.sda1.in0 disk.dm-1.in0) proc.memory.availableMB 1816.22265625* proc.memory.percent19.046878814697266 cpu25.34537815334417 disk.drbd1260.out0.671875 disk.dm-0.in 10055.546875 disk.sda3.in0 disk.sda3.out136.6875 disk.sda2.out0 proc.cpu.threads17% disk./.usageGB1150.9617233276367' disk.nvme0n1.out100299.25732421875$ disk.dm-0.out100299.25732421875 disk.sda.in0 disk.sda.out136.6875" disk.nvme0c0n1.in 10055.546875*(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts228.143' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent45.628600000000006 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ߓ(:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp47.0 gpu.0.powerWatts320.214' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.0428 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0Cm(: Ƥ disk.nvme0n1.in 10055.546875 disk.dm-0.out 100656.078125 disk.sda.in0 network.sent 5.936069e+06 disk.sda1.in0 disk.sda3.out 136.9765625" disk.nvme0c0n1.in 10055.546875$ disk.nvme0c0n1.out 100656.078125 disk.dm-1.in0 disk.drbd1260.out0.671875( proc.memory.availableMB 1816.2109375 disk.dm-0.in 10055.546875 disk.sda3.in0 disk.sda.out 136.9765625 disk.sda2.out0 disk.dm-1.out 136.9765625! proc.memory.rssMB 3120.640625 cpu25.345626497744583 proc.cpu.threads17) disk./.usagePercent77.27556290527016% disk./.usageGB1151.4617309570312" disk.nvme0n1.out 100656.078125 disk.sda1.out0 disk.sda2.in0 disk.drbd1260.in0$ memory_percent88.91472816467285* proc.memory.percent19.046878814697266 network.recv 1.42902e+06k#Q(:  gpu.0.gpu98.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts249.58' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent49.916000000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0b'(:  disk.drbd1260.out 0.67578125# disk.dm-0.out100978.6630859375 disk.sda1.in0 disk.sda.in0 disk.sda2.in0 disk.sda2.out0 network.recv 1.432875e+06) disk./.usagePercent77.27561845890509$ disk./.usageGB1151.462558746338! disk.nvme0n1.in 10144.3203125 disk.dm-0.in 10144.3203125 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 137.32421875( disk.nvme0c0n1.out100978.6630859375 disk.dm-1.out 137.32421875 disk.drbd1260.in0& disk.nvme0n1.out100978.6630859375 disk.sda.out 137.32421875$ memory_percent88.91613483428955! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266 network.sent 5.941741e+06) proc.memory.availableMB 1815.98046875 cpu25.345701307325225 proc.cpu.threads17# disk.nvme0c0n1.in 10144.0234375 disk.dm-1.in0x2- _wandbruntime4463  _runtime44636k _runtime4463.409224223 _step204 $ lr-AdamW/pg10.00084328533874082 $ lr-AdamW/pg20.00084328533874082 trainer/global_step2549 % _timestamp1.7884695733686926e+09 trainer/global_step2549 % _timestamp1.7884695733686926e+09 _runtime4463.409224223 _step204 $ lr-AdamW/pg10.00084328533874082 $ lr-AdamW/pg20.000843285338740822- _wandbruntime4463  _runtime4463{>O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed561842375181312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated120635400594432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7555249569280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated539205562214873 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated561844916972544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated36720290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak328 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 5920228621312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated47959033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed11306678 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5909938643821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated18201824 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed533293083774672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated533293109202640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7555190629888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed29508043 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 _runtime4463.411099158 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current462 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated11306681 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed36720284 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed561842375181312 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated36720290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated47959033 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed36720284 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed84659633 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed47939349 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed120635312557056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated84679323 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 trainer/global_step2549 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated555924688351232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed18201365 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed555924662923264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current459 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed555924662923264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated29508505 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5917712258048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed47939349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed113080121927168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated561844916972544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 % _timestamp1.7884695733699389e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed561842375181312 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated36720290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated47959033 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed36720284 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed84659633 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed47939349 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed120635312557056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak 3142753280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated84679323 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak 37060870144 trainer/global_step2549 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak 40265318400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated555924688351232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed18201365 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed555924662923264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current459 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed555924662923264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated29508505 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak 3140736376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 5917712258048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed47939349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed113080121927168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated561844916972544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated3674 % _timestamp1.7884695733699389e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed 5909938643821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated18201824 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current88037376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed533293083774672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated533293109202640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7555190629888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed29508043 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 _runtime4463.411099158 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak3674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 25143684122- _wandbruntime4464  _runtime4464 r'M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed84691865 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak1203 _step206 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed84691865 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak 35901791232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed556135653298688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated47977410 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated113124212394496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated3674 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5919987474432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated36734171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated120682367863808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed539407823382931 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5922503851008 B8DeviceStatsMonitor.on_train_batch_end/active.all.current19716 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated84711581 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed29519237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated84711581 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak23981 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed29519237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak3674 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current2146 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak 36938011465 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak 37060870144 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current 2541804544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed556135653298688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak1528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated47977410 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak 3142753280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated113124212394496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak 3140736376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated 37060870144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current3674 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current 2539796588 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak 40265318400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current19710 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current 2516376576 D;DeviceStatsMonitor.on_train_batch_endeV/segment.all.allocated3674 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current 2516376576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed 5919987474432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated36734171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated120682367863808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed539407823382931 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak 38561404416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated 5922503851008"/i2- _wandbruntime4464  _runtime4464E Y 0loss/FRt1/assignment_loss1.4455529161625438 /loss/FRt2/assignment_loss3.760378532750266 2loss/SRqqt4/assignment_loss1.6254016405769758 .loss/FRt1/detection_loss2.145145427021715 epoch4 1loss/FBt4/assignment_loss0.19506452972365712 %loss/total_loss45.68331004269865 _step207 0loss/FBt1/assignment_loss0.1417590368785927 2loss/SRqqt2/assignment_loss1.7394957037668832 0loss/SRqqt2/detection_loss2.496346416187958 trainer/global_step2549 0loss/FBt2/assignment_loss0.3587404254823923 .loss/FBt1/detection_loss0.693464844736313 /loss/FBt3/detection_loss3.0110046236138595 _runtime4464.887991435 0loss/FRt3/assignment_loss3.3623381898953366 1loss/FBt3/assignment_loss0.20246108258020526 /loss/FRt2/detection_loss2.6031321457454135 /loss/FRt4/detection_loss2.1667158505706876 1loss/SRqqt4/detection_loss1.5882137692400387 /loss/FBt4/detection_loss2.2647696923125875 /loss/FRt3/detection_loss2.7188331154676586 1loss/SRqqt1/detection_loss1.4127958402922838 0loss/SRqqt3/detection_loss2.080741870146926 $ _timestamp1.788469574847644e+09 0loss/FRt4/assignment_loss2.9637543045812182 1loss/SRqqt1/assignment_loss1.239947725530013 1loss/SRqqt3/assignment_loss2.047206685587924 /loss/FBt2/detection_loss3.4200456738471985$^ 2loss/SRqqt2/assignment_loss1.7394957037668832 0loss/SRqqt2/detection_loss2.496346416187958 trainer/global_step2549 0loss/FBt2/assignment_loss0.3587404254823923 .loss/FBt1/detection_loss0.693464844736313 /loss/FBt3/detection_loss3.0110046236138595 _runtime4464.887991435 0loss/FRt3/assignment_loss3.3623381898953366 1loss/FBt3/assignment_loss0.20246108258020526 /loss/FRt2/detection_loss2.6031321457454135 /loss/FRt4/detection_loss2.1667158505706876 1loss/SRqqt4/detection_loss1.5882137692400387 /loss/FBt4/detection_loss2.2647696923125875 /loss/FRt3/detection_loss2.7188331154676586 1loss/SRqqt1/detection_loss1.4127958402922838 0loss/SRqqt3/detection_loss2.080741870146926 $ _timestamp1.788469574847644e+09 0loss/FRt4/assignment_loss2.9637543045812182 1loss/SRqqt1/assignment_loss1.239947725530013 1loss/SRqqt3/assignment_loss2.047206685587924 /loss/FBt2/detection_loss3.4200456738471985 0loss/FRt1/assignment_loss1.4455529161625438 /loss/FRt2/assignment_loss3.760378532750266 2loss/SRqqt4/assignment_loss1.6254016405769758 .loss/FRt1/detection_loss2.145145427021715 epoch4 1loss/FBt4/assignment_loss0.19506452972365712 %loss/total_loss45.68331004269865 _step207 0loss/FBt1/assignment_loss0.14175903687859279Z(: ٠ gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp48.0 gpu.0.powerWatts322.458' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent64.4916 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ِ(: ش" disk.nvme0n1.in10178.35546875 disk.sda.in0 disk.sda.out 137.9453125 disk.sda2.out0! proc.memory.rssMB 3120.640625 cpu25.345905765916886 proc.cpu.threads17 disk.drbd1260.in0 disk.drbd1260.out 0.67578125 disk.sda1.out0 disk.sda3.in0 disk.sda2.in0$ disk.nvme0c0n1.in10178.35546875 disk.dm-1.in0* proc.memory.percent19.046878814697266 network.sent 5.947331e+06 network.recv 1.437035e+06% disk./.usageGB1151.6694717407227 disk.dm-0.in10178.35546875 disk.sda1.in0( disk.nvme0c0n1.out101510.9833984375 disk.dm-1.out 137.9453125$ memory_percent88.91804218292236) disk./.usagePercent77.28950456356982& disk.nvme0n1.out101510.9833984375# disk.dm-0.out101510.9833984375 disk.sda3.out 137.9453125) proc.memory.availableMB 1815.66796875iUSR(:  disk.drbd1260.in0( proc.memory.availableMB 1815.9140625! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266% disk./.usageGB1151.7107772827148 disk.sda3.out 138.43359375$ memory_percent88.91654014587402 cpu25.346080194228104 proc.cpu.threads17 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda.in0 disk.sda2.in0 disk.dm-1.out 138.43359375 disk.drbd1260.out 0.6796875 network.recv 1.442171e+06) disk./.usagePercent77.29227661315058" disk.nvme0n1.in10178.35546875 disk.dm-0.in10178.35546875$ disk.dm-0.out101858.55322265625 disk.sda2.out0) disk.nvme0c0n1.out101858.55322265625 disk.dm-1.in0 network.sent 6.03404e+06' disk.nvme0n1.out101858.55322265625 disk.sda.out 138.43359375$ disk.nvme0c0n1.in10178.35546875a(:  gpu.0.gpu97.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts227.888' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent45.577600000000004 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0ρ(:  disk.dm-1.out 138.8984375 disk.drbd1260.out 0.6796875 cpu25.346389515911802 proc.cpu.threads17 network.sent 6.041909e+06% disk./.usageGB1151.7102661132812 disk.sda.in0 disk.sda2.in0$ disk.nvme0c0n1.in10178.35546875 disk.drbd1260.in0* proc.memory.percent19.046878814697266) disk./.usagePercent77.29224230814098" disk.nvme0n1.in10178.35546875 disk.dm-0.in10178.35546875$ disk.dm-0.out102139.16259765625 disk.sda3.in0 disk.sda3.out 138.8984375 disk.sda.out 138.8984375 disk.sda2.out0 network.recv 1.446777e+06' disk.nvme0n1.out102139.16259765625 disk.sda1.in0 disk.sda1.out0) disk.nvme0c0n1.out102139.16259765625# memory_percent88.9181137084961& proc.memory.availableMB 1815.65625! proc.memory.rssMB 3120.640625 disk.dm-1.in0j&(:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts231.106' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent46.2212 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0u̓(:  gpu.0.gpu96.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts239.923' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent47.9846 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0z(:  network.sent 6.04551e+06 network.recv 1.450499e+06 disk.sda1.in0 disk.sda1.out0 disk.sda.in0 disk.sda2.out0$ disk.nvme0c0n1.in10178.35546875( proc.memory.availableMB 1815.6640625% disk./.usageGB1151.6867980957031 disk.dm-0.out102281.2421875 disk.sda3.out 139.31640625 disk.sda2.in0 disk.dm-1.in0 disk.dm-1.out 139.31640625 disk.drbd1260.out 0.68359375! proc.memory.rssMB 3120.640625) disk./.usagePercent77.29066734979034# disk.nvme0n1.out102281.2421875 disk.dm-0.in10178.35546875 disk.sda3.in0* proc.memory.percent19.046878814697266 proc.cpu.threads17" disk.nvme0n1.in10178.35546875 disk.sda.out 139.31640625% disk.nvme0c0n1.out102281.2421875 disk.drbd1260.in0$ memory_percent88.91806602478027 cpu25.34667698177047Mw(: 欌! disk.dm-0.out102699.51171875 disk.sda1.out0 disk.sda3.out 139.640625# disk.nvme0c0n1.in 10407.3671875& disk.nvme0c0n1.out102699.51171875 disk.drbd1260.in0$ memory_percent88.91656398773193 network.sent 6.051227e+06% disk./.usageGB1151.6868782043457 disk.sda1.in0 disk.dm-1.in0! proc.memory.rssMB 3120.640625* proc.memory.percent19.046878814697266 proc.cpu.threads17$ disk.nvme0n1.out102699.51171875 disk.dm-0.in 10407.3671875 disk.sda.out 139.640625 disk.sda2.out0 cpu25.346814263072833) disk./.usagePercent77.29067272594857! disk.nvme0n1.in 10407.3671875 disk.sda3.in0 disk.sda.in0 disk.sda2.in0 disk.dm-1.out 139.640625 disk.drbd1260.out 0.68359375) proc.memory.availableMB 1815.91015625 network.recv 1.454804e+06(:  gpu.0.gpu95.0 gpu.0.memory17+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp46.0 gpu.0.powerWatts234.295' gpu.0.enforcedPowerLimitWatts500.0) gpu.0.powerPercent46.858999999999995 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0](:  gpu.0.gpu98.0 gpu.0.memory18+ gpu.0.memoryAllocated47.51960754394531* gpu.0.memoryAllocatedBytes 40819032064 gpu.0.temp43.0 gpu.0.powerWatts190.661' gpu.0.enforcedPowerLimitWatts500.0 gpu.0.powerPercent38.1322 gpu.0.smClock1410 gpu.0.memoryClock1593! gpu.0.correctedMemoryErrors0# gpu.0.uncorrectedMemoryErrors0{|<|(: Ė" disk.dm-0.out102885.712890625 disk.sda.out 140.08203125) proc.memory.availableMB 1815.91015625 disk.sda2.out0% disk.nvme0n1.out102885.712890625 disk.dm-0.in 10407.3671875 cpu25.347044254921805 network.recv 1.458271e+06# disk.nvme0c0n1.in 10407.3671875' disk.nvme0c0n1.out102885.712890625 disk.dm-1.in0 disk.drbd1260.in0 disk.drbd1260.out0.6875$ memory_percent88.91656398773193! proc.memory.rssMB 3120.640625 network.sent 6.058153e+06 disk.sda1.in0 disk.sda1.out0 disk.sda3.in0 disk.sda3.out 140.08203125 disk.sda.in0* proc.memory.percent19.046878814697266 proc.cpu.threads17) disk./.usagePercent77.29120419758965% disk./.usageGB1151.6947975158691 disk.sda2.in0 disk.dm-1.out 140.08203125! disk.nvme0n1.in 10407.3671875'2- _wandbruntime4545  _runtime4545tE % lr-AdamW/pg10.000827834912300903 % lr-AdamW/pg20.000827834912300903 trainer/global_step2599 % _timestamp1.7884696551295707e+09 _runtime4545.170143798 _step208X1 % lr-AdamW/pg10.000827834912300903 % lr-AdamW/pg20.000827834912300903 trainer/global_step2599 % _timestamp1.7884696551295707e+09 _runtime4545.170143798 _step208vj2- _wandbruntime4545  _runtime4545aH'O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed566474181694464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed549443097812401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed543419519253872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current 40265318400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed572505682599936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed115283190394368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated37414340 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak2684 =7DeviceStatsMonitor.on_train_batch_start/num_device_free0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak 3142753280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated86294275 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current468 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current2146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed48860251 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current 2541791232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak23682 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated115283219492352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated572508224391168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak 35901791232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated 6034017268736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated11523281 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current 3204448256 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated48879935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed18548288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak 1400896000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak1528 _runtime4545.175081713 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak 960105472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed0 trainer/global_step2599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current25427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated 37060870144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed0 % _timestamp1.7884696551309223e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed 7700990714368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed37414334 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated566474207122432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed30071566 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated572508224391168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed37414334 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed48860251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed122984181108736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 6026092926941 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _step209 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed572505682599936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated48879935 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated543419544681840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 6031500905472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed11523278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated18548756 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7701049653760 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated122984269146112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current 2541791232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed 6031500905472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated566474207122432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed566474181694464 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak 38561404416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak23981 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current29097984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak 36938011465 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current 2516363264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak 451453440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated549445637608781 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated1528 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current1528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak 35901791232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current 37060870144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak 3204448256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current19690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current19684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current19690 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak23981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed37414334 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated566474207122432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed30071566 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak 34275006464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current 2514368412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak1203 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak23682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current58939392 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current 2516363264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated572508224391168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed37414334 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed48860251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed122984181108736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated 6026092926941 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated 40265318400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak2146 _step209 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak 38561404416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed572505682599936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc3674 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current 2539796380 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current3674 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated48879935 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated543419544681840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed 6031500905472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed11523278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated18548756 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated 7701049653760 60DeviceStatsMonitor.on_train_batch_start/num_ooms0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated 3204448256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak2684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated122984269146112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections0dO2- _wandbruntime4546  _runtime4546q~M B